Gait analysis and diagnosis method and device, electronic equipment and readable storage medium

Abstract

The invention provides a gait analysis and diagnosis method and device, electronic equipment and a readable storage medium, and relates to the field of artificial intelligence. The method comprises the following steps of: performing analysis and judgment according to the specific condition of a tested object to obtain a proper data sampling method, recording the tested object by using a monocular camera or a multi-view camera system, predicting a three-dimensional key point coordinate track of the motion of the tested object by using a visual model corresponding to the shooting method, and determining the three-dimensional key point coordinate track according to the predicted three-dimensional key point coordinate track. And performing motion reconstruction by using the coordinate track of the key point to obtain a diagnosis basis, and performing comprehensive analysis and diagnosis on the tested object through an artificial intelligence algorithm.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a gait analysis and diagnosis method, device, electronic device, and readable storage medium. Background Art

[0002] Gait disorder is an important clinical manifestation of many cranial nerve diseases. However, the early manifestation of this symptom is easily overlooked, especially when the symptoms appear in elderly patients, they are often mistaken for the natural manifestation of physical aging. If the root cause of the disease is not taken seriously and treated in time, the gait disorder will continue to develop and worsen, causing patients to have more serious clinical symptoms such as tremor and gait freezing, and even causing patients to fall frequently. Early detection of these symptoms and timely medical treatment can help doctors intervene more effectively. Effective control in the early stages of the disease can reduce treatment costs and improve patient recovery. In addition, for patients who already have more serious gait disorders, rehabilitation training should be carried out under the guidance of medical staff. Quantification and evaluation of the effect of rehabilitation training during the treatment process is very important for treatment. By evaluating the patient's recovery, it can help doctors adjust the training plan in time.

[0003] At present, the clinical assessment of gait disorders is mainly carried out through various scale methods. The most commonly used screening scale is the Wisconsin Gait Scale. Medical staff need to receive professional training before they can correctly use the above scale assessment method. The evaluation process is complicated, time-consuming, and occupies a large amount of medical resources. Moreover, since such scales need to be completed through the inquiry and observation of others, the evaluation results of different people vary greatly, making it difficult to achieve objective data quantification, resulting in large errors in the final evaluation results.

[0004] In order to achieve an objective evaluation of gait function, smart devices are also used in clinical practice to record and evaluate the patient's gait. However, such equipment is relatively expensive and complicated to use, often requiring patients to wear cumbersome electronic devices, including motion sensors, electrocardiogram, electroencephalogram, and electromyography recording devices. Medical staff need to complete the debugging and installation of the equipment before use, and require patients to cooperate with repeated debugging, which brings a great burden to patients. On the other hand, wearing electronic devices for patients will cause unnatural gait problems for patients, affecting the accuracy of the evaluation results. Summary of the invention

[0005] The present application provides a gait analysis and diagnosis system based on artificial intelligence, which makes full use of artificial intelligence algorithms to provide a convenient gait analysis and diagnosis system for medical institutions and experiments. It has the advantages of simple operation process, no need for subjects to wear any equipment, and can perform data sampling by themselves, which is convenient for data follow-up during long-term treatment. The system of the present application provides two data collection methods according to different application scenarios, and has data storage and management functions, which can compare and analyze historical data of different subjects during long-term treatment.

[0006] In a first aspect, a gait analysis and diagnosis method is provided, the method comprising:

[0007] Analyze and judge based on the specific situation of the subjects to come up with an appropriate data sampling method.

[0008] The subject is recorded using a monocular camera or a multi-camera system, and the three-dimensional coordinate trajectory of key points of the subject's movement is predicted using a visual model corresponding to the shooting method.

[0009] The body feature parameters of the subjects are extracted through the deep vision model, and the motion is reconstructed using the key point coordinate trajectory to obtain the basis for diagnosis.

[0010] The subjects are comprehensively analyzed and diagnosed through artificial intelligence algorithms.

[0011] In a possible implementation of the first aspect, analysis and judgment are performed according to the specific situation of the subject, including:

[0012] The physical condition and medical characteristics of the subjects, as well as the type of exercise to be tested.

[0013] The data collection methods are divided into monocular camera sampling, where the subjects can use electronic devices with shooting functions such as mobile phones to take self-service photos according to the requirements of medical staff; and multi-camera system sampling, where the subjects need to perform motion tests and shooting under the guidance of medical staff in a venue where a camera array is installed.

[0014] In a possible implementation of the first aspect, using a visual model corresponding to the shooting method to predict a three-dimensional key point coordinate trajectory of the subject's movement includes:

[0015] The 3D key point prediction model is used to extract the key point trajectory of the subject in the camera coordinate system from the monocular video.

[0016] The 2D key point prediction model is used to first extract the 2D key point trajectories from multiple cameras, and then the 3D key point trajectories are reconstructed using the 2D key point trajectories.

[0017] In a possible implementation of the first aspect, body feature parameters of the subject are extracted through a deep vision model, where the body features include a body contour grid and joint spacing length parameters.

[0018] In an optional implementation, motion reconstruction using key point coordinate trajectories includes:

[0019] A character model with the same parameters is constructed in a simulation environment according to the body characteristic parameters of the subject.

[0020] Using the optimization algorithm, the motion trajectory of the character model is reconstructed in the simulation environment, so that it moves according to the three-dimensional trajectory of the key points extracted by the visual model.

[0021] In a possible implementation of the first aspect, the two-dimensional key point prediction model may be a pre-trained model with fixed network weights or a network weight model retrained using a specific training sample set, including:

[0022] The visual model is trained based on the training sample set.

[0023] A plurality of groups of training samples are obtained from a training sample set, each of which is a plurality of frames of sample images carrying annotation information, and the annotation information is the correct coordinates of key points of each frame of the sample images in the plurality of frames of sample images.

[0024] Each set of training samples is input into the visual model to obtain the prediction results of each set of training samples.

[0025] According to the prediction results of each group of training samples and the annotation information of each group of training samples, the model parameters of the visual model are adjusted.

[0026] When the preset training conditions are met, the training process of the visual model ends.

[0027] In a possible implementation manner of the first aspect, the preset training conditions include: the number of training times reaches a preset number of times or the output results of each group of training samples reach a preset accuracy.

[0028] In a possible implementation of the first aspect, extracting a key point trajectory of a subject in a camera coordinate system from a monocular video includes:

[0029] The visual model is used to obtain the two-dimensional key point coordinates of the subject.

[0030] The camera depth coordinates of the key points are estimated based on the 2D key point coordinates and the kinematic parameters of the human body structure.

[0031] Reconstruct the three-dimensional coordinates of the key points based on the two-dimensional key point coordinates and the key point coordinates in the depth direction.

[0032] In a second aspect, a gait analysis and diagnosis device is provided, the device comprising:

[0033] The data acquisition unit is used to shoot the subject. Depending on different situations, the acquisition unit is a single-camera shooting unit or a multi-camera shooting unit. The subject's whole body can be observed in the camera's field of view.

[0034] The prediction unit is used to predict the three-dimensional spatial motion trajectory of the key points of the subject.

[0035] The motion reconstruction unit is used to reconstruct the three-dimensional motion of the character model that complies with physical constraints in the simulation environment according to the three-dimensional spatial motion trajectory of the key points of the subject.

[0036] The storage unit is used to store the data generated by each test for comparative analysis during long-term treatment and for training the analytical diagnosis unit.

[0037] The analysis and diagnosis unit is used to analyze the reconstructed motion and make a diagnostic evaluation on the subject.

[0038] According to a third aspect, an electronic device is provided, including:

[0039] A memory, a processor, and a computer program stored in the memory and executable on the processor execute the method in the above first aspect or any possible implementation of the first aspect.

[0040] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the computer program is used to execute the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a framework diagram of a gait analysis and diagnosis system based on artificial intelligence provided in an embodiment of the present application;

[0042] Figure 2 This is an example of a feasible shooting area of ​​a multi-camera system provided in an embodiment of the present application;

[0043] Figure 3 It is a motion reconstruction method applicable to both monocular camera and multi-camera shooting methods provided in the embodiment of the present application;

[0044] Figure 4 The embodiment of the present application provides a method for using clinical indicators obtained during motion reconstruction for diagnostic analysis;

[0045] Figure 5 It is a component unit of the overall system provided by the embodiment of the present application;

[0046] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0048] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0049] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0050] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0051] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0052] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0053] The present application provides a gait analysis and diagnosis system based on artificial intelligence, which is used to quantify and analyze diagnostic indicators for patients with gait disorders caused by neurological diseases. The patient's limb movement characteristics are important diagnostic indicators for neurological diseases such as multiple sclerosis and Alzheimer's disease, which can help doctors judge the patient's condition and recovery, and then choose appropriate treatment methods. However, the process of acquiring limb movement characteristics is complicated and difficult to quantify. Existing quantification methods generally require high-cost electronic hardware systems and require patients to wear professional sensing equipment and collect data under the guidance of medical staff. In this case, patients wearing electronic sensors will be affected by the wearable devices, which makes the movement unnatural, and the collected motion data is different from that without wearing the device, which makes the analysis results have errors.

[0054] On the other hand, after obtaining the patient's movement data, medical staff need to manually compare and analyze to evaluate the patient's condition. This evaluation process is inevitably affected by the medical staff's subjective influence, resulting in inaccurate evaluation and affecting the doctor's judgment of the patient's true condition.

[0055] In view of this, the present application proposes a diagnostic system that is convenient for medical staff and patients to use, using artificial intelligence technology to replace the traditional system. The diagnostic system provided by the present application avoids the requirement for expensive hardware and reduces the equipment cost on the one hand, and on the other hand, does not require the patient to wear any device, making the movements during motion collection more natural and reflecting the patient's real situation. In addition, through deep learning technology, the collected patient motion data is analyzed to obtain accurate quantitative indicators to assist doctors in making judgments and treatments.

[0056] An example of an artificial intelligence-based gait analysis method provided in this application will be described in detail below.

[0057] Figure 1 This is a framework diagram of an artificial intelligence-based gait analysis and diagnosis system provided in an embodiment of the present application. The first step is to judge the specific situation of the subject and select an appropriate data sampling method:

[0058] If the patient's condition is mild and he or she can walk easily, he or she can choose to go to a medical institution regularly to record in a specially built multi-camera shooting system.

[0059] If the patient's condition is serious and it is inconvenient for him to go back and forth to medical institutions frequently, he can choose to use a mobile device with a camera function and have his family assist in recording under the guidance of medical staff. Then send the video to the medical institution and use this application system for data analysis.

[0060] Among them, the multi-camera shooting system consists of more than two cameras, and can reconstruct the target motion more accurately through imaging of cameras in different orientations.

[0061] The data obtained by patients using a monocular camera for self-photography lacks information in the depth direction and needs to be estimated through artificial intelligence algorithms, so the reconstruction results may contain errors.

[0062] Both shooting methods require that all body parts of the subject be clearly visible in the camera's field of view, that there be no obstructions during the shooting process, and that the camera remain still and not move.

[0063] In particular, the multi-camera system requires that the subject is fully visible in each camera's view angle, and before shooting, the shooting site needs to be planned according to the camera's view angle and placement. Figure 2 The shooting system shown is composed of six cameras, all of which are evenly distributed along the circumference. The feasible area of ​​the entire shooting system is the intersection of all camera angles. There are no special requirements for the camera arrangement method. Ensure that the subject is clearly visible in each camera without obstruction. The number of cameras N is greater than 2 to meet the requirements.

[0064] In terms of human motion reconstruction, the present application embodiment provides an accurate motion reconstruction method based on artificial intelligence, which is applicable to the above two data sampling methods. Figure 3 The figure shows the process of the human body motion reconstruction method provided in the embodiment of the present application.

[0065] After obtaining the motion video of the subject, the first step is to use a computer vision model to extract the three-dimensional motion trajectory of the key points of the photographed target. In some embodiments of the present application, the three-dimensional motion trajectory extracted by the monocular video is predicted in the camera coordinate system, so the depth direction information and the real scaling ratio are missing. The real body parameters of the subject, such as height, arm length, leg length, etc., need to be input into the model to construct the character model in the simulation, so as to accurately reconstruct the real movement of the character.

[0066] The three-dimensional motion trajectory extracted from multi-view video is more accurate, without loss of information in any dimension, and the coordinates of the three-dimensional key points can be used to estimate the real body parameters of the subject.

[0067] Furthermore, after obtaining the three-dimensional key points of the subject, motion reconstruction is performed by performing kinematic constraints and dynamic constraints on the character model in the simulation, so that the three-dimensional coordinates of the key points of the character model fit the three-dimensional motion trajectory extracted from the video to the greatest extent, while conforming to the physical laws in the real world, such as Newton-Euler motion formulas and friction angle constraints in friction collision processes.

[0068] In some embodiments of the present application, the joint angles of the character model in the initial simulation are used as free variables, the above two constraints are constructed using mathematical expressions, the kinematic terms and the dynamic terms are minimized simultaneously in the form of a loss function, and the final numerical optimization results of the free variables are used as the motion reconstruction results of the subject. In the simulation, the three-dimensional motion trajectory of the character is visualized and presented to the doctor.

[0069] In some embodiments of the present application, Figure 3 As shown, in the motion reconstruction process, the character model is first subjected to kinematic constraints, and then to dynamic constraints. The step-by-step process can improve the convergence speed of the optimization algorithm. Some embodiments of the present application use the Gauss-Newton method to optimize the target.

[0070] When using a multi-camera system to predict 3D key points, you can use the calibration board and calibration program provided by AprilTag to calibrate the relative positions of multiple cameras. Further, use the 2D key point prediction model to annotate the single frame of the video shot by each camera to obtain the 2D coordinates of the key points in the image. Then, use the triangulation principle and the relative position relationship between the above cameras to reconstruct the key points in 3D. Obtain the accurate 3D motion trajectory of the key points.

[0071] In some embodiments of the present application, by applying kinematic constraints and dynamic constraints to the character model in the simulation, important reference indicators for clinical diagnosis can be quantified, such as Figure 4 shown.

[0072] The clinical indicators obtained through motion reconstruction include kinematic characteristics and dynamic characteristics. The embodiment of the present application can obtain key kinematic clinical parameters such as step length, step frequency, stride, etc. by reconstructing the motion of the character model in a simulation environment. It can also obtain dynamic parameters such as joint torque and plantar pressure. The above parameters are multimodally fused, and the patient's condition and recovery degree are quantitatively analyzed using a deep neural network model. In the long-term treatment process, by comparing the indicators obtained from each diagnosis, it can help doctors make timely adjustments to the treatment methods.

[0073] In some embodiments of the present application, the above-mentioned deep neural network uses MLP (multi-layer perceptron network), first normalizes the input data, and then sends it to the network to obtain the prediction result. The network model is trained by supervised learning.

[0074] In some embodiments of the present application, the storage unit saves the sampled data of each patient, which is used for comparison during long-term treatment and for continuously updating the above neural network prediction model. As clinical data continues to increase, the model continuously updates weights through supervised learning, and the prediction results will become more accurate.

[0075] The system components provided in the embodiment of the present application are as follows: Figure 5 As shown, two implementation modes of the data acquisition unit are the above-mentioned single-camera or multi-camera shooting system, which are used to obtain the motion data of the subject. The prediction unit is the above-mentioned deep visual model, which is used to extract the key point trajectory from the shot video. The motion reconstruction unit reconstructs the character model in a simulation environment through kinematic constraints and dynamic constraints, so that the key points of the character model fit the key point trajectory extracted by the prediction unit, and at the same time conform to the constraints of Newtonian mechanics. The reconstruction result includes the kinematic motion characteristics generated by the above-mentioned kinematic constraint optimization and the dynamic characteristic parameters generated by the above-mentioned dynamic constraint optimization. Furthermore, the physical characteristic parameters of the subject are multimodally fused, and the recovery situation is quantitatively analyzed by the analysis and diagnosis unit. The storage unit is used to save the patient's historical data for comparison, to adjust the treatment process, and to update the analysis and prediction unit, so that the model continues to evolve and become more accurate.

[0076] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: a memory 601, a processor 602, and a computer program stored in the memory 601 and executable on the processor 602.

[0077] In some embodiments of the present application, the memory 601, the processor 602 and the communication interface 603 are implemented independently. The communication interface 603, the memory 601 and the processor 602 are connected and communicated via a bus. The bus can be an industrial standard architecture bus, a peripheral device interconnection bus or an extended industrial standard architecture bus, etc.

[0078] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed by a processor, they are used to implement the technical solution of any reconstruction or analysis method in the embodiment of the present application.

[0079] Although the present application is disclosed in terms of preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A gait analysis and diagnosis method, It is characterized in that The method comprises: Analyze and judge the specific situation of the subjects to come up with a suitable data sampling method; Using a monocular camera or a multi-camera system to record the subject, and using a visual model corresponding to the shooting method to predict the three-dimensional key point coordinate trajectory of the subject's movement; Extracting the body feature parameters of the subject through a deep vision model, and performing motion reconstruction using the key point coordinate trajectory to obtain a diagnosis basis; The subjects are comprehensively analyzed and diagnosed through artificial intelligence algorithms.

2. The gait analysis and diagnosis method according to claim 1, It is characterized in that Analyze and judge based on the specific situation of the subject, including: The physical condition and disease characteristics of the subject, and the type of exercise to be tested; The data collection method is divided into monocular camera sampling, in which the subject uses an electronic device with a shooting function to perform self-service shooting according to the requirements of medical staff; and multi-camera system sampling, in which the subject needs to perform motion testing and shooting under the guidance of medical staff in a test site where a camera array is installed.

3. The gait analysis and diagnosis method according to claim 1, It is characterized in that Predicting the three-dimensional key point coordinate trajectory of the subject's movement using a visual model corresponding to the shooting method includes: Using a deep learning-based 3D key point prediction model, the key point trajectory of the subject in the camera coordinate system is extracted from the monocular video; Using a two-dimensional key point prediction model based on deep learning, two-dimensional key point trajectories are first extracted from videos shot by multiple camera systems, and then the three-dimensional key point trajectories are reconstructed using the two-dimensional key point trajectories.

4. The gait analysis and diagnosis method according to claim 1, It is characterized in that The body feature parameters of the subject are extracted through a deep vision model, and the body features include a body contour grid and joint spacing length parameters.

5. The gait analysis and diagnosis method according to claim 1, It is characterized in that The step of performing motion reconstruction using the key point coordinate trajectory includes: Constructing a character model with the same parameters in a simulation environment according to the body characteristic parameters of the subject; The motion trajectory of the character model is reconstructed in a simulation environment by using an optimization algorithm so that the character model can move by fitting the three-dimensional trajectory of key points extracted by the visual model.

6. The method according to claim 3, It is characterized in that The two-dimensional key point prediction model is a pre-trained model with fixed network weights or a network weight model retrained using a specific training sample set, and the method includes: Acquire multiple groups of training samples in the training sample set, each group of training samples in the multiple groups of training samples is a multiple-frame sample shot image carrying annotation information, and the annotation information is the correct coordinates of key points of each frame of the sample shot image in the multiple frames of sample shot images; Inputting each set of training samples into the visual model to obtain prediction results of each set of training samples; According to the prediction results of each group of training samples and the labeling information of each group of training samples, the model parameters of the visual model are adjusted by the gradient descent principle of the neural network; When the preset training condition is reached, the training process of the two-dimensional key point prediction model is ended.

7. The method according to claim 3, It is characterized in that The step of extracting the key point trajectory of the subject in the camera coordinate system from the monocular video includes: Using a two-dimensional key point prediction model to obtain the two-dimensional key point coordinates of the subject in the photographed sample; According to the two-dimensional key point coordinates and human body structure kinematic parameters, the key point camera depth direction coordinates are estimated through optimization algorithm and joint angle constraints; The three-dimensional coordinates of the key points are reconstructed according to the two-dimensional key point coordinates and the estimated values ​​of the key point coordinates in the depth direction.

8. A gait analysis and diagnosis device, It is characterized in that The device comprises: A shooting unit, used to shoot the subject. Depending on different situations, the shooting unit is a single-camera shooting unit or a multi-camera shooting unit. The subject can be observed in its entire body in the camera's field of view. A prediction unit, used to predict the three-dimensional spatial motion trajectory of the key points of the subject; A reconstruction unit, used for reconstructing the three-dimensional motion of the character model that complies with physical constraints in a simulation environment according to the three-dimensional spatial motion trajectory of the key points of the subject; The analysis and diagnosis unit is used to analyze the reconstructed motion and make a diagnostic evaluation on the subject.

9. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that The instruction is executed by the processor to implement the method described in any one of the technical solutions of claims 1-7.

Images