A child gait abnormality evaluation auxiliary system based on human posture detection

Abstract

The application discloses a kind of children gait abnormality evaluation auxiliary systems based on human posture detection, comprising: video acquisition module, for obtaining video;Video contains gait image when child walks, and video does not limit shooting scene and shooting equipment;Human posture detection module, for detecting the human body detection frame of each human body object in video and the human body key point in frame using preset human posture detection algorithm;Human tracking module, for frame-by-frame matching the human body detection frame of each human body object in video, and according to the human body key point in matched human body detection frame, generate the continuous gait feature of human body object;Gait evaluation module, for comparing and analyzing the continuous gait feature of human body object with corresponding gait feature reference standard, and outputting the auxiliary evaluation result whether human body object exists gait abnormality according to analysis result.The application can accurately and efficiently evaluate children gait abnormality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical devices, and particularly relates to a child gait abnormality evaluation auxiliary system based on human posture detection. BACKGROUND

[0002] Gait abnormality of preschool children is one of the most common reasons for children orthopedic clinic visits. The causes of gait abnormality of preschool children are very diverse, including posture, inflammation, trauma, deformity, tumor and other causes, diagnosis is very difficult, often leading to misdiagnosis or missed diagnosis.

[0003] In clinical practice, imageological examination methods are often used to identify whether the gait of children is abnormal, including ultrasound, X-ray, magnetic resonance and CT (Computed Tomography) examination, etc. However, such imageological examination methods have potential radioactive hazards and cannot be used for a long time and frequently, which is not conducive to tracking the condition of the sick children.

[0004] In addition to imageological examination methods, doctors can also identify gait abnormalities of children based on their personal experience. However, due to differences in body shape and / or walking posture of different patients, and due to the uneven experience of different doctors, it is difficult to form a quantitative identification standard for identifying whether the gait of children is abnormal based on the personal experience of doctors. In addition, the manual identification method needs the doctor to see the sick child, and the execution efficiency is relatively low, and the economic and time cost is high.

[0005] In the prior art, there is a method of human posture evaluation by means of image analysis, which pastes reflective markers on the joint parts of the human body, then makes the person walk in a dedicated shooting area, and uses multiple cameras to shoot the human body, collects the activity track of the joint parts of the person during walking, and judges the current posture of the human body by analyzing the activity track collected from the images shot by multiple cameras. However, this method requires the person to wear special markers, and requires a special shooting site (requires multiple cameras). In addition, this method usually only allows one person to appear in the lens. However, in the scene of identifying gait abnormalities of children, the child's family members and other adults often need to guide the child, that is, multiple people will inevitably appear in the lens. Therefore, for all the above reasons, the human posture evaluation method has not been practically applied to the scene of identifying gait abnormalities of children. SUMMARY

[0006] In order to solve the above problems existing in the prior art, the present application provides a child gait abnormality evaluation auxiliary system based on human posture detection.

[0007] The technical problem to be solved by the present application is solved by the following technical scheme:

[0008] An auxiliary system for evaluating gait abnormality of children based on human posture detection, comprising:

[0009] a video acquisition module configured to acquire a video, wherein the video contains gait images of children walking, and the video does not limit the shooting scene and the shooting device;

[0010] a human posture detection module configured to detect human bounding boxes of each human object in the video and human key points in the bounding boxes by using a preset human posture detection algorithm, wherein the human objects include the children;

[0011] a human tracking module configured to match the human bounding boxes of each human object in the video frame by frame, and generate continuous gait features of the human objects according to the human key points in the matched human bounding boxes;

[0012] a gait evaluation module configured to compare and analyze the continuous gait features of the human objects with corresponding gait feature reference standards, and output an auxiliary evaluation result of whether the human objects have gait abnormality according to an analysis result.

[0013] Optionally, the human tracking module matches the human bounding boxes of each human object in the video frame by frame, comprising:

[0014] predicting a human bounding box of a current frame from a human bounding box of a previous frame by using a Kalman filtering method, as a first human bounding box of the current frame;

[0015] taking a human bounding box of the current frame actually detected by the human posture detection algorithm as a second human bounding box, and determining the matched first human bounding box and the second human bounding box by using a preset human bounding box matching process;

[0016] for the matched second human bounding box, associating the second human bounding box to a human object corresponding to the matched first human bounding box;

[0017] for the unmatched second human bounding box, associating the second human bounding box to a new human object;

[0018] if there is an unmatched first human bounding box, counting a number of times that the human object corresponding to the first human bounding box has no second human bounding box associated therewith until the current time, and deleting the human object corresponding to the first human bounding box if the number of times is greater than an upper limit;

[0019] continuing to process a next frame.

[0020] Optionally, the human bounding box matching process comprises:

[0021] The ReID network is used to calculate a cosine distance of an in-frame image of each pair of the first human body bounding box and the second human body bounding box, and to calculate a Mahalanobis distance of each pair of the first human body bounding box and the second human body bounding box;

[0022] According to the cosine distance calculation result and the Mahalanobis distance calculation result, each pair of the first human body bounding box and the second human body bounding box whose cosine distance and Mahalanobis distance are less than respective thresholds is screened out;

[0023] The Hungarian algorithm is used to perform secondary screening on each pair of the first human body bounding box and the second human body bounding box screened out, to obtain matched pairs of the first human body bounding box and the second human body bounding box.

[0024] Optionally, the human body bounding box matching process further comprises:

[0025] After obtaining the matched pairs of the first human body bounding box and the second human body bounding box, for a second human body bounding box that is not matched successfully, an intersection-over-union of the second human body bounding box and each first human body bounding box that is not matched successfully is calculated; if there is a first human body bounding box that is not matched successfully and a second human body bounding box that is not matched successfully, and the intersection-over-union of the second human body bounding box and the first human body bounding box is greater than a preset intersection-over-union, it is determined that the second human body bounding box and the first human body bounding box are matched.

[0026] Optionally, the human-machine interaction module is further configured to receive second input information selected by a user for an analysis mode; the analysis mode comprises a front analysis mode or a side analysis mode.

[0027] The human-machine interaction module is configured to receive first input information selected by a user for a human body object, and trigger the gait evaluation module for the human body object selected by the user according to the first input information.

[0028] Optionally, the human-machine interaction module is further configured to receive second input information selected by a user for an analysis mode; the analysis mode comprises a front analysis mode or a side analysis mode.

[0029] The human body tracking module generates continuous gait features of the human body object according to the human body key points in the matched human body bounding box, comprising:

[0030] According to the human body key points in the matched human body bounding box, each item of human body posture parameters corresponding to the analysis mode selected by the user is calculated frame by frame.

[0031] The continuous gait features of the human body object are generated according to each item of human body posture parameters calculated frame by frame.

[0032] Optionally, each item of human body posture parameters corresponding to the front analysis mode comprises:

[0033] An angle between bilateral shoulders and a horizontal direction, and / or an angle between bilateral hip joints and the horizontal direction, and / or an angle between bilateral knee joints and the horizontal direction, and / or an angle between bilateral ankle joints and the horizontal direction

[0034] The formula for calculating the included angle is:

[0035] wherein y R , y L , x R , x L respectively represent the y coordinates of the right key point, the y coordinates of the left key point, the x coordinates of the right key point, and the x coordinates of the left key point of the bilateral shoulder, the bilateral hip joint, the bilateral knee joint, or the bilateral ankle joint, and a is the included angle.

[0036] Optionally, the human body posture parameters corresponding to each side mode include a hip joint flexion angle, and / or a knee joint flexion angle, and / or an ankle joint flexion angle.

[0037] The formula for calculating the hip joint flexion angle includes:

[0038] or,

[0039]

[0040] wherein X A is the x coordinate of the key point of the hip joint, Y A is the y coordinate of the key point of the hip joint, X B is the x coordinate of the key point of the knee joint, Y B is the y coordinate of the key point of the knee joint, a L is the hip joint flexion angle of the left side, a R is the hip joint flexion angle of the right side.

[0041] The formula for calculating the knee joint flexion angle is:

[0042]

[0043] wherein, represents a vector from the key point of the hip joint to the key point of the knee joint, represents a vector from the key point of the knee joint to the key point of the ankle joint; and a2 is the knee joint flexion angle.

[0044] The formula for calculating the ankle joint flexion angle is:

[0045]

[0046] wherein, represents a vector from the key point of the ankle joint to the key point of the toe, represents a vector from the key point of the ankle joint to the key point of the knee joint; and a3 is the ankle joint flexion angle.

[0047] Optionally, the continuous gait features include a plurality of items, each of which is generated by a human posture parameter calculated frame by frame.

[0048] The gait assessment module compares the continuous gait features of the human object with the corresponding gait feature reference standard to obtain an auxiliary evaluation result of whether the human object has gait abnormalities, including:

[0049] For each human posture parameter, the time distribution curve formed by the human posture parameter calculated frame by frame is compared with the reference standard curve corresponding to the human posture parameter; and whether the human object has gait abnormalities is determined according to the analysis result.

[0050] Optionally, the way of comparing the time distribution curve with the reference standard curve includes:

[0051] Peak and valley information of the time distribution curve is extracted; the peak and valley information includes a wave crest and a wave trough;

[0052] Statistical average peak and valley information of the time distribution curve is determined according to the extracted peak and valley information;

[0053] The statistical average peak and valley information is compared with the peak and valley information of the reference standard curve.

[0054] In the child gait abnormality evaluation auxiliary system based on human posture detection provided by the application, the human posture detection algorithm is used in the human posture detection module to detect the human detection frame of each human object in the video and the human key points in the frame, without the need to paste reflective markers on the human joint parts, and without the requirement for the shooting scene. Then, the human detection frame of each human object in the video is matched frame by frame in the human tracking module to distinguish different human objects, so that the continuous gait features of each human object can be generated according to the human key points in the matched human detection frame of each human object. In this way, even if there are multiple people in the video, the child to be determined whether to have gait abnormalities can be effectively distinguished. Then, the gait assessment module compares the continuous gait features of the human object with the corresponding gait feature reference standard to obtain an auxiliary evaluation result of whether the child has gait abnormalities; this analysis process uses a unified identification standard and does not require human intervention, and can qualitatively and quantitatively give a useful auxiliary evaluation result of whether the child gait has abnormalities.

[0055] In addition, since the shooting scene and the shooting device of the video are not required, the video can be shot in advance and input into the system to quickly obtain the auxiliary evaluation result, and the efficiency is also relatively high.

[0056] The application will be further described in detail below with reference to the accompanying drawings and the application. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a structural schematic diagram of a child gait abnormality evaluation auxiliary system provided by an embodiment of the present application;

[0058] Figure 2 is a workflow diagram of a child gait abnormality evaluation auxiliary system provided by an embodiment of the present application;

[0059] Figure 3 shows a human key point involved in a child gait abnormality evaluation auxiliary system provided by an embodiment of the present application;

[0060] Figure 4 shows a graphical interface in a child gait abnormality evaluation auxiliary system provided by an embodiment of the present application;

[0061] Figure 5 shows another graphical interface in a child gait abnormality evaluation auxiliary system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0062] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0063] In order to accurately and efficiently evaluate child gait abnormality, an embodiment of the present application provides a child gait abnormality evaluation auxiliary system based on human posture detection. In actual application, the system provided by the embodiment of the present application can be a desktop computer, a portable computer, a smart mobile terminal, a medical device or a health care device, etc. electronic device. The electronic device contains a storage medium, which can store a computer program. It also contains a processor that can run the above computer program. The electronic device is also provided with an external communication interface to import external video into the system through the communication interface.

[0064] The above communication interface includes but is not limited to Bluetooth interface, USB interface and / or network interface, etc.

[0065] The above electronic device can also be integrated or associated with a keyboard, a mouse, an operation panel and / or a display, etc. module, so as to carry out human-computer interaction. In addition, the electronic device can also be integrated or associated with a camera, so that the electronic device can be directly used to shoot video.

[0066] It should be noted that the embodiment of the present application does not limit the specific type of the above electronic device.

[0067] The child gait abnormality evaluation auxiliary system based on human posture detection provided by an embodiment of the present application will be described in detail below. Figure 1 shows a structural schematic diagram of a child gait abnormality evaluation auxiliary system provided by an embodiment of the present application; Figure 2 A workflow diagram of the child gait abnormality evaluation auxiliary system provided by the embodiment of the present application is shown.

[0068] In combination Figure 1 And Figure 2 As shown in the figure, the child gait abnormality evaluation auxiliary system provided by the embodiment of the present application comprises a video acquisition module, a human posture detection module, a human tracking module, and a gait evaluation module.

[0069] The video acquisition module is configured to acquire a video, wherein the video contains gait images of a child walking, and the video does not limit the shooting scene and the shooting device. For example, the video can be shot by a mobile phone in a place familiar to the child, such as a bedroom or a living room, so that the child's cooperation degree will be relatively high.

[0070] In actual application, the video acquisition module can be a video shooting module, such as a camera or a video camera. Alternatively, the video acquisition module can be a video import module, such as a USB interface module, a Bluetooth interface module, or a network interface module.

[0071] The human posture detection module is configured to detect a human detection frame of each human object in the video and a human key point in the frame by using a preset human posture detection algorithm.

[0072] It can be understood that the each human object mentioned here includes a child to be confirmed whether there is a gait abnormality; in addition to the child, the human objects in the video can also include a doctor or a relative of the child.

[0073] Human posture estimation is a research field in the field of computer vision, which detects human bodies in the input image and generates corresponding key points by analyzing the image, and obtains the motion posture of the human body by analyzing the coordinate motion track of the human key points. Human posture estimation algorithm can analyze specific actions for different observation objects, can observe different action events, and can intelligently identify human key points (including human shoulder, elbow, wrist, hip, knee, and ankle). By using these human key points, the moving speed of the human body in space, the joint, and the trunk balance can be determined.

[0074] There are many existing human posture estimation algorithms, mainly including regression method and heat map method. The regression method refers to an algorithm that analyzes the input picture and directly outputs the key point coordinates; while the heat map method generates a heat map carrying key point information by algorithm, and the value in the heat map represents the probability value of the key point being located at this position. The algorithm obtains the key point information of the human body by selecting the position with the maximum probability value in the heat map.

[0075] The inventor found that the gait analysis method based on the openpose algorithm has low accuracy and takes a long time when used for human posture detection, and the gait analysis method based on the openpose algorithm needs to use a high-speed camera for capture, the device is expensive, not easy to popularize, and the device is large in size and not easy to carry, which brings many inconveniences in actual use and does not match the use scenario of the embodiment of the present application.

[0076] Therefore, in order to balance the detection accuracy and the detection speed, the human posture detection algorithm used in the embodiment of the present application includes: kapao or yolov7-pose. Here, kapao and yolov7-pose are both existing human posture detection algorithms (also called models), which are algorithms based on deep neural networks, and can both detect targets in a video, output human key points of the target in the video, and return a human detection box of the target in the video; for a single target, the human key points of the target are located in the human detection box of the target, Figure 3 Each human key point involved in the embodiment of the present application is shown. Based on the human detection box returned by kapao or yolov7-pose, human tracking can be achieved, so as to distinguish different human objects in the video.

[0077] The human tracking module is configured to match the human detection boxes of each human object in the video frame by frame, and generate continuous gait features of the human object according to the human key points in the matched human detection box.

[0078] Specifically, the human tracking module matches the human detection boxes of each human object in the video frame by frame, including:

[0079] (1) predicting the human detection box of the current frame from the human detection box of the previous frame by using the Kalman filtering method, as the first human detection box of the current frame;

[0080] (2) taking the human detection box of the current frame actually detected by the human posture detection algorithm as the second human detection box, and determining the matched first human detection box and the second human detection box by using a preset human detection box matching process;

[0081] (3-a) for the second human detection box that is successfully matched, associating the second human detection box to the human object corresponding to the first human detection box matched with the second human detection box;

[0082] (3-b) for the second human detection box that is not successfully matched, associating the second human detection box to a new human object;

[0083] (3-c) If there is a first human body bounding box that is not successfully matched, count the number of times the human body object corresponding to the first human body bounding box is not associated with a second human body bounding box up to the current time, and if the number is greater than an upper limit, delete the human body object corresponding to the first human body bounding box;

[0084] (4) Then continue to process the next frame.

[0085] The steps (3-a) to (3-b) above can be executed in parallel. The occurrence of step (3-c) can be caused by temporary occlusion or movement of the human body object out of the shooting scene.

[0086] In the above step (2), the human body bounding box matching process can include:

[0087] (A) Using a ReID (Person Re-identification, pedestrian re-identification) network to calculate the cosine distance of the image in each pair of first human body bounding box and second human body bounding box, and to calculate the Mahalanobis distance of each pair of first human body bounding box and second human body bounding box;

[0088] (B) According to the cosine distance calculation result and the Mahalanobis distance calculation result, screening each pair of first human body bounding box and second human body bounding box whose cosine distance and Mahalanobis distance are less than the respective threshold value;

[0089] (C) Using the Hungarian algorithm to perform secondary screening on each pair of first human body bounding box and second human body bounding box screened out, to obtain the matched pairs of first human body bounding box and second human body bounding box.

[0090] Here, the Hungarian algorithm (Hungarian algorithm) is an algorithm for finding the maximum matching in graph theory.

[0091] In addition, after obtaining the matched pairs of first human body bounding box and second human body bounding box in step (C), the human body bounding box matching process can further include:

[0092] For the second human body bounding box that is not successfully matched, calculate the intersection-over-union of the second human body bounding box and each first human body bounding box that is not successfully matched; if there is an intersection-over-union greater than a preset intersection-over-union, determine that the second human body bounding box and the first human body bounding box for which the intersection-over-union is calculated are matched.

[0093] The human body tracking module generates continuous gait features of the human body object according to the human body key points in the matched human body bounding box, including: calculating each item of human body posture parameter frame by frame according to the human body key points in the matched human body bounding box. Since multiple items of human body posture parameters can be calculated for each frame, each item of human body posture parameter calculated frame by frame can generate an item of continuous gait feature.

[0094] When the image of the human object in the frame is a side view, the human posture parameters calculated by the human tracking module can include a hip joint flexion angle, and / or a knee joint flexion angle, and / or an ankle joint flexion angle.

[0095] The calculation formula of the hip joint flexion angle includes:

[0096] Or,

[0097]

[0098] wherein X A is an x coordinate of a key point of the hip joint, Y A is a y coordinate of the key point of the hip joint, X B is an x coordinate of a key point of the knee joint, Y B is a y coordinate of the key point of the knee joint; a L is a hip joint flexion angle of the left side, a R is a hip joint flexion angle of the right side.

[0099] The calculation formula of the knee joint flexion angle is:

[0100]

[0101] wherein, represents a vector of a key point of the hip joint pointing to a key point of the knee joint, represents a vector of a key point of the knee joint pointing to a key point of the ankle joint; a2 is a knee joint flexion angle.

[0102] The calculation formula of the ankle joint flexion angle is:

[0103]

[0104] wherein, represents a vector of a key point of the ankle joint pointing to a key point of the toe, represents a vector of a key point of the ankle joint pointing to a key point of the knee joint; a3 is an ankle joint flexion angle.

[0105] In the embodiment of the present application, for the human object in a side view in the video, not only the posture feature at the ankle joint related to walking is considered, but also the posture features at the shoulder joint, the hip joint and the like in the walking process are further considered, so that the posture features of any one or more joints can be used to evaluate whether the human body has gait abnormality.

[0106] When the image of the human object in the frame is a front view, the human posture parameters calculated by the human body tracking module can include: an angle between the bilateral shoulders and the horizontal direction (also referred to as a shoulder joint inclination angle), and / or an angle between the bilateral hip joints and the horizontal direction (also referred to as a hip joint inclination angle), and / or an angle between the bilateral knee joints and the horizontal direction (also referred to as a knee joint inclination angle), and / or an angle between the bilateral ankle joints and the horizontal direction (also referred to as an ankle joint inclination angle).

[0107] wherein the calculation formula of the angle is:

[0108] In the formula, y R , y L , x R , x L respectively represent the y coordinate of the right key point, the y coordinate of the left key point, the x coordinate of the right key point, and the x coordinate of the left key point of the bilateral shoulders, the bilateral hip joints, the bilateral knee joints, or the bilateral ankle joints, and a represents the angle.

[0109] Compared with the above-mentioned method of analyzing only the unilateral gait, the embodiment of the present application considers the difference between the bilateral gaits of the human object in the front view in the video, so as to identify whether the human object has gait abnormalities according to the difference between the bilateral gaits, which can be more accurate than the result of evaluation relying on unilateral gait analysis.

[0110] The gait evaluation module is configured to compare and analyze the continuous gait features of the human object with the corresponding gait feature reference standard, and output an auxiliary evaluation result of whether the human object has gait abnormalities according to the analysis result.

[0111] Specifically, the gait evaluation module compares and analyzes the time distribution curve formed by the human posture parameter calculated frame by frame with the reference standard curve corresponding to the human posture parameter for each human posture parameter, so as to determine whether the human object has gait abnormalities according to the analysis result.

[0112] The comparison and analysis of the time distribution curve and the reference standard curve in the gait evaluation module can include:

[0113] (1) extracting peak and valley information of the time distribution curve; the peak and valley information includes a wave peak and a wave valley;

[0114] (2) determining statistical average peak and valley information of the time distribution curve according to the extracted peak and valley information;

[0115] (3) comparing the statistical average peak and valley information with the peak and valley information of the reference standard curve.

[0116] Specifically, the difference between the wave crest in the statistical average peak-valley information and the wave crest in the reference standard curve is calculated, and the difference between the wave trough in the statistical average peak-valley information and the wave trough in the reference standard curve is calculated, and if the difference of one item exceeds the corresponding threshold, it indicates that the human subject may have gait abnormalities; otherwise, it indicates that the human subject has no gait abnormalities.

[0117] In addition, when the comparative analysis is performed on multiple human posture parameters, if the wave crest / wave trough of the peak-valley information of one human posture parameter exceeds the corresponding threshold, it indicates that the human subject may have gait abnormalities; otherwise, it indicates that the human subject has no gait abnormalities.

[0118] In actual application, the time distribution curve of the human posture parameters of healthy children can be collected to establish a normal walking database of children in different age groups, so that the reference standard curve of children in different age groups can be formulated based on the database.

[0119] It should be noted that the auxiliary evaluation result output in the embodiment of the present application is only for the reference of doctors, and plays an auxiliary role, and the auxiliary evaluation result output in the embodiment of the present application should not be used as a medical diagnosis result.

[0120] In the child gait abnormality evaluation auxiliary system based on human posture detection provided by the embodiment of the present application, the human posture detection algorithm is used in the human posture detection module to detect the human detection frame of each human subject in the video and the human key points in the frame, without the need of pasting reflective markers on the human joint parts, and without the requirement for the shooting scene. Then, the human detection frames of each human subject in the video are matched frame by frame in the human tracking module to distinguish different human subjects, so that the continuous gait features of each human subject can be generated according to the human key points in the matched human detection frame of each human subject. In this way, even if there are multiple people in the video, the child to be determined whether to have gait abnormalities can be effectively distinguished. Then, the continuous gait features of the human subject are compared and analyzed with the corresponding gait feature reference standard in the gait evaluation module, and the auxiliary evaluation result of whether the child has gait abnormalities can be obtained; this analysis process adopts a unified identification standard and does not need human intervention, and can quantitatively give a useful auxiliary evaluation result of whether the child gait has abnormalities.

[0121] In addition, since the embodiment of the present application does not require the shooting scene and the shooting device of the video, the video can be shot in advance and input into the system to quickly obtain the auxiliary evaluation result, and the efficiency is also relatively high.

[0122] Optionally, the child gait abnormality evaluation auxiliary system based on human posture detection provided by the embodiment of the present application can further include a human-computer interaction module.

[0123] The human-computer interaction module is configured to receive first input information of a user selecting a human body object, and trigger the gait assessment module for the human body object selected by the user according to the first input information.

[0124] The human-computer interaction module can interact with the user through a graphical interface. For example, the user clicks the clicking options of each human body object provided by the human-computer interaction module through a mouse. Alternatively, in another implementation manner, the user can also input the id of each human body object through a keyboard (for example, the id of the child in FIG. 6B). Figure 4

[0125] After receiving the first input information, the human-computer interaction module outputs the auxiliary assessment result of whether the human body object specified in the first input information has a gait abnormality.

[0126] In another implementation manner, the gait assessment module can perform comparison and analysis in the background, and save the analysis result of each human body object. When the user inputs the first input information, the gait assessment module correspondingly outputs the auxiliary assessment result of whether the human body object selected by the user has a gait abnormality.

[0127] Optionally, the human-computer interaction module is further configured to receive second input information of a user selecting an analysis mode, wherein the analysis mode selected by the user includes a front analysis mode or a side analysis mode.

[0128] Here, the human-computer interaction module receives the second input information in a similar manner, and thus details are not repeated.

[0129] Correspondingly, when generating the continuous gait features of the human body object according to the human body key points in the matched human body detection frame, the human body tracking module can calculate each human body posture parameter corresponding to the analysis mode selected by the user frame by frame according to the human body key points in the matched human body detection frame, and then generate the continuous gait features of the human body object according to each human body posture parameter calculated frame by frame.

[0130] Optionally, the child gait abnormality assessment auxiliary system based on human body posture detection provided by the embodiment of the present application can further include a display module.

[0131] The display module is configured to display a video labeled with the human body detection frame and the human body key points detected by the human body posture detection module, display the continuous gait features generated by the human body tracking module, and display the comparison and analysis process data and the auxiliary assessment result of the gait assessment module.

[0132] Figure 4 FIG. 6B shows a graphical interface provided by the display module. As shown in FIG. 6B, the user selects the id of the child through the human-computer interaction module, and then the display module displays the video labeled with the human body detection frame and the human body key points detected by the human body posture detection module. Figure 5 ​The human body detection frame of the child to be selected by the user is distinguished from the human body detection frames of other people by different colors. In addition Figure 4 The graphical interface also includes a region (a region in which analysis results in the region in the graphical interface are displayed) for displaying auxiliary evaluation results. Figure 4

[0133] The comparative analysis process data can include the time distribution curve, the reference standard curve, effects of the two being plotted in the same coordinate system, a difference between peak and valley information of the two, and a corresponding threshold, and the like.

[0134] In addition, a timing of displaying each item of content is related to a timing of the user interacting with the system through the human-computer interaction module, and embodiments of the present application do not limit the timing of displaying each item of content. Embodiments of the present application also do not limit a display form of each item of content, as long as the displayed content is presented to the user through the display.

[0135] In summary, the child gait abnormality evaluation auxiliary system provided by embodiments of the present application first uses a human pose estimation method to perform gait analysis on a child with gait abnormality, and performs gait abnormality evaluation through gait features.

[0136] The child gait abnormality evaluation auxiliary system provided by embodiments of the present application comprehensively considers walking gait features of a child from multiple angles, can analyze a walking gait of the child from multiple angles, and provides a multi-dimensional evaluation means, thereby helping a doctor to improve a diagnosis accuracy.

[0137] The child gait abnormality evaluation auxiliary system provided by embodiments of the present application has no emission hazard, can be used for evaluation multiple times during a rehabilitation training period, and can provide accurate reference information for a doctor, thereby helping the doctor to track treatment.

[0138] The child gait abnormality evaluation auxiliary system provided by embodiments of the present application has a simple and easy-to-operate graphical interface, is convenient to operate, and has high evaluation efficiency.

[0139] The child gait abnormality evaluation auxiliary system provided by embodiments of the present application can extract a key target from a child gait video in which there is interference from other people, accurately detect the target, and has a relatively small requirement for a shooting scene, and has high robustness.

[0140] The child gait abnormality evaluation auxiliary system provided by embodiments of the present application has a relatively simple composition, has a low requirement for hardware configuration, and can obtain a qualified video only by using a conventional smart phone when shooting. Therefore, a common computer or a common mobile phone can implement the child gait abnormality evaluation auxiliary system provided by embodiments of the present application, thereby being convenient to use and being capable of quickly performing analysis and evaluation.

[0141] ​The application further provides a computer readable storage medium. A computer program is stored in the computer readable storage medium, and the computer program is executed by a processor to implement the following method steps:

[0142] (1) obtaining a video; the video contains gait images of children walking, and the video does not limit the shooting scene and the shooting device;

[0143] (2) detecting a human body detection frame of each human body object in the video and a human body key point in the frame by using a preset human body posture detection algorithm; the human body object includes the child;

[0144] (3) matching the human body detection frame of each human body object in the video frame by frame, and generating continuous gait features of the human body object according to the human body key point in the matched human body detection frame;

[0145] (4) comparing and analyzing the continuous gait features of the human body object with corresponding gait feature reference standards, and outputting an auxiliary evaluation result of whether the human body object has gait abnormalities according to the analysis result.

[0146] Optionally, matching the human body detection frame of each human body object in the video frame by frame comprises:

[0147] predicting the human body detection frame of the current frame by using a Kalman filtering method on the human body detection frame of the previous frame, as a first human body detection frame of the current frame;

[0148] taking the human body detection frame of the current frame actually detected by using the human body posture detection algorithm as a second human body detection frame, and determining the matched first human body detection frame and the second human body detection frame by using a preset human body detection frame matching process;

[0149] for the matched second human body detection frame, associating the second human body detection frame to the human body object corresponding to the matched first human body detection frame;

[0150] for the unmatched second human body detection frame, associating the second human body detection frame to a new human body object;

[0151] if there is an unmatched first human body detection frame, counting the number of times that the human body object corresponding to the first human body detection frame does not have a second human body detection frame associated therewith, and if the number of times is greater than an upper limit, deleting the human body object corresponding to the first human body detection frame;

[0152] continuing to process the next frame.

[0153] Optionally, the human body detection frame matching process comprises:

[0154] The ReID network is used to calculate a cosine distance of an in-frame image of each pair of the first human body detection frame and the second human body detection frame, and to calculate a Mahalanobis distance of each pair of the first human body detection frame and the second human body detection frame;

[0155] According to the cosine distance calculation result and the Mahalanobis distance calculation result, each pair of the first human body detection frame and the second human body detection frame whose cosine distance and Mahalanobis distance are less than respective thresholds is screened out;

[0156] The Hungarian algorithm is used to perform secondary screening on each pair of the first human body detection frame and the second human body detection frame screened out, to obtain matched pairs of the first human body detection frame and the second human body detection frame.

[0157] Optionally, the human body detection frame matching process can further include:

[0158] After obtaining the matched pairs of the first human body detection frame and the second human body detection frame, for a second human body detection frame that is not matched successfully, an intersection-over-union ratio of the second human body detection frame and each first human body detection frame that is not matched successfully is calculated; if there is a first human body detection frame that is not matched successfully and a second human body detection frame that are matched, the intersection-over-union ratio of the second human body detection frame and the first human body detection frame is calculated.

[0159] Optionally, the computer program-implemented method steps can further include:

[0160] Receiving first input information of a user selecting a human body object; and performing the step of comparing and analyzing the continuous gait feature of the human body object with the corresponding gait feature reference standard according to the first input information, and outputting the auxiliary evaluation result of whether the human body object has gait abnormalities according to the analysis result.

[0161] Optionally, the computer program-implemented method steps can further include:

[0162] Receiving second input information of a user selecting an analysis mode; the analysis mode includes: a front analysis mode or a side analysis mode;

[0163] The continuous gait feature of the human body object generated according to the human body key points in the matched human body detection frame includes:

[0164] According to the human body key points in the matched human body detection frame, each item of human body posture parameters in the analysis mode selected by the user is calculated frame by frame;

[0165] The continuous gait feature of the human body object is generated according to each item of human body posture parameters calculated frame by frame.

[0166] Optionally, each item of human body posture parameters in the front mode includes:

[0167] An angle between the bilateral shoulder and the horizontal direction, and / or an angle between the bilateral hip joint and the horizontal direction, and / or an angle between the bilateral knee joint and the horizontal direction, and / or an angle between the bilateral ankle joint and the horizontal direction

[0168] The angle is calculated by the following formula:

[0169] Wherein, y R , y L , x R , x L respectively represent the y coordinate of the right key point, the y coordinate of the left key point, the x coordinate of the right key point, and the x coordinate of the left key point of the bilateral shoulder, the bilateral hip joint, the bilateral knee joint or the bilateral ankle joint, and a is the angle.

[0170] Optionally, the human body posture parameters corresponding to each side mode include: a hip joint flexion angle, and / or a knee joint flexion angle, and / or an ankle joint flexion angle.

[0171] The calculation formula of the hip joint flexion angle includes:

[0172] Or,

[0173]

[0174] Wherein, X A is the x coordinate of the key point of the hip joint, Y A is the y coordinate of the key point of the hip joint, X B is the x coordinate of the key point of the knee joint, Y B is the y coordinate of the key point of the knee joint; a L is the hip joint flexion angle of the left side, a R is the hip joint flexion angle of the right side.

[0175] The calculation formula of the knee joint flexion angle is:

[0176]

[0177] Wherein, represents a vector from the key point of the hip joint to the key point of the knee joint, represents a vector from the key point of the knee joint to the key point of the ankle joint; a2 is the knee joint flexion angle.

[0178] The calculation formula of the ankle joint flexion angle is:

[0179]

[0180] Wherein, a vector pointing from the key point representing the ankle joint to a key point representing the knee joint; and a vector pointing from the key point representing the ankle joint to a key point representing the knee joint; and

[0181] Optionally, the continuous gait features include a plurality of items, each of which is generated by a human posture parameter calculated frame by frame.

[0182] The comparison of the continuous gait features of the human object with the corresponding gait feature reference standard includes:

[0183] For each human posture parameter, the time distribution curve formed by the human posture parameter calculated frame by frame is compared with the reference standard curve corresponding to the human posture parameter; and whether the human object has gait abnormality is determined according to the analysis result.

[0184] Optionally, the comparison of the time distribution curve with the reference standard curve includes:

[0185] The peak-valley information of the time distribution curve is extracted; the peak-valley information includes a wave crest and a wave trough.

[0186] The statistical average peak-valley information of the time distribution curve is determined according to the extracted peak-valley information.

[0187] The statistical average peak-valley information is compared with the peak-valley information of the reference standard curve.

[0188] In still another embodiment of the present application, a computer program product containing instructions, which, when run on a computer, causes the computer to perform the method steps of the method implemented by the computer program described above.

[0189] It should be noted that, for the storage medium / computer program product embodiment, since it is basically similar to the system embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the system embodiment.

[0190] It should be noted that the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0191] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate different embodiments or examples described in the specification.

[0192] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and implemented by those skilled in the art through viewing the drawings and the disclosure.

[0193] The above is a further detailed description of the present application in conjunction with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as falling within the scope of protection of the present application.

Claims

1. A child gait abnormality evaluation assistance system based on human posture detection, characterized by, include: A video acquisition module is used to acquire a video; the video includes a gait image of a child walking, and the video is not limited to the shooting scene and shooting equipment; A human posture detection module is used to detect the human body detection frame of each human object in the video and the human body key points within the frame using a preset human posture detection algorithm; The individual human subjects include the child; A human tracking module is used to match the human detection frame of each human object in the video frame by frame, and generate continuous gait features of the human object based on the human key points in the matched human detection frame; A gait assessment module is used to compare and analyze the continuous gait characteristics of the human subject with the corresponding gait characteristic reference standard, and output an auxiliary assessment result of whether the human subject has gait abnormality based on the analysis result; The human body tracking module matches the human body detection frame of each human object in the video frame by frame, including: The human body detection frame of the previous frame is used to predict the human body detection frame of the current frame using the Kalman filter method as the first human body detection frame of the current frame; The human body detection frame of the current frame actually detected by the human posture detection algorithm is used as the second human body detection frame, and a preset human body detection frame matching process is used to determine the matching first human body detection frame and second human body detection frame; For the successfully matched second human body detection frame, associating the second human body detection frame with the human body object corresponding to the first human body detection frame matched with the second human body detection frame; For the second human body detection frame that is not successfully matched, associating the second human body detection frame with a new human body object; If there is an unmatched first human detection frame, count the number of times the human object corresponding to the first human detection frame has not been associated with a second human detection frame up to the current time. If the number is greater than an upper limit, delete the human object corresponding to the first human detection frame. Continue processing the next frame; The human body detection frame matching process includes: The ReID network is used to calculate the cosine distance of the images within each pair of the first human detection frame and the second human detection frame, and the Mahalanobis distance of each pair of the first human detection frame and the second human detection frame; According to the cosine distance calculation results and the Mahalanobis distance calculation results, each pair of the first human detection frame and the second human detection frame whose cosine distance and Mahalanobis distance are both smaller than the respective thresholds is selected; The Hungarian algorithm is used to perform secondary screening on the screened pairs of the first human detection frame and the second human detection frame to obtain matching pairs of the first human detection frame and the second human detection frame; The human body detection frame matching process also includes: After obtaining each matched pair of the first human detection frame and the second human detection frame, for the second human detection frame that was not successfully matched, the intersection-and-union ratio of the second human detection frame and each unmatched first human detection frame is calculated; if the intersection-and-union ratio is greater than the preset intersection-and-union ratio, it is determined that the second human detection frame for which the intersection-and-union ratio is calculated matches the first human detection frame. 2.The child gait abnormality evaluation assistance system based on human posture detection according to claim 1, wherein Also includes: Human-computer interaction module; The human-computer interaction module is configured to receive first input information from a user selecting a human body object; According to the first input information, the gait assessment module is triggered for the human object selected by the user. 3.The child gait abnormality evaluation assistance system based on human posture detection according to claim 2, wherein The human-computer interaction module is further configured to receive second input information of a user selecting an analysis mode; the analysis mode includes a front analysis mode or a side analysis mode. The human body tracking module generates continuous gait features of the human body object according to the human body key points in the matched human body detection frame, including: According to the human body key points in the matched human body detection frame, each item of human body posture parameters corresponding to the analysis mode selected by the user is calculated frame by frame; According to each item of human body posture parameters calculated frame by frame, the continuous gait features of the human body object are generated. 4.The child gait abnormality evaluation assistance system based on human posture detection according to claim 3, wherein The human body posture parameters corresponding to the front analysis mode include: The included angles between the bilateral shoulders and the horizontal direction, and / or the included angles between the bilateral hip joints and the horizontal direction, and / or the included angles between the bilateral knee joints and the horizontal direction, and / or the included angles between the bilateral ankle joints and the horizontal direction The formula for calculating the included angle is: ; wherein, , , , respectively represent the y coordinates of the right key point of the bilateral shoulder or the bilateral hip joint or the bilateral knee joint or the bilateral ankle joint, the y coordinates of the left key point, the x coordinates of the right key point, the x coordinates of the left key point, is the included angle. 5.The child gait abnormality evaluation assistance system based on human posture detection of claim 3, wherein The human body posture parameters corresponding to the side analysis mode include: the hip joint flexion angle, and / or the knee joint flexion angle, and / or the ankle joint flexion angle. The calculation formula of the hip joint flexion angle includes: or, alternatively, ； wherein, the coordinates of the key point of the hip joint, x the coordinates of the key point of the hip joint, y the coordinates of the key point of the knee joint, x the coordinates of the key point of the knee joint, y the hip joint flexion angle of the left side, the hip joint flexion angle of the right side;​​​​ The calculation formula of the knee joint flexion angle is: ； wherein, represents a vector from a key point of the hip joint to a key point of the knee joint, represents a vector from a key point of the knee joint to a key point of the ankle joint; is a knee flexion angle; The calculation formula of the ankle joint flexion angle is: ； wherein, represents a vector from the key point of the ankle joint to the key point of the foot tip, represents a vector from the key point of the ankle joint to the key point of the knee joint; is the ankle flexion angle. 6.The child gait abnormality evaluation assistance system based on human posture detection according to claim 3, wherein The continuous gait features include multiple items, each of which is generated by one item of human body posture parameters calculated frame by frame. The gait evaluation module compares and analyzes the continuous gait features of the human body object with the corresponding gait feature reference standard to obtain an auxiliary evaluation result of whether the human body object has gait abnormalities, including: For each item of human body posture parameters, the time distribution curve formed by the item of human body posture parameters calculated frame by frame is compared and analyzed with the reference standard curve corresponding to the item of human body posture parameters; and whether the human body object has gait abnormalities is determined according to the analysis result. 7.The child gait abnormality evaluation assistance system based on human posture detection according to claim 6, wherein The way of comparing and analyzing the time distribution curve with the reference standard curve includes: Extracting peak-valley information of the time distribution curve; the peak-valley information includes a wave peak and a wave valley; Determining statistical average peak-valley information of the time distribution curve according to the extracted peak-valley information; Comparing the statistical average peak-valley information with the peak-valley information of the reference standard curve.

Images