Intelligent gait analysis auxiliary diagnosis system for knee joint function

Abstract

The invention relates to the technical field of medical treatment, and discloses an intelligent gait analysis auxiliary diagnosis system for knee joint functions. Comprising a data acquisition module, a 3D human body posture modeling module, a gait feature extraction module, a gait feature measurement and analysis module, a deep learning analysis and diagnosis module, an interpretive model and user interaction module, a system integration module and a data management and storage module. The data acquisition module acquires human body gait video data through a multi-view camera system. A non-contact multi-view camera shooting technology and an artificial intelligence algorithm are adopted, the dependence on expensive equipment and professional operation is reduced, the deployment and maintenance cost of the system is reduced, medical services related to gait analysis can be carried out in medium and small hospitals, the step of pasting mark points on patients is omitted through a non-contact data acquisition mode, and the medical service efficiency is improved. The operation process is simplified, the dependence on professional medical staff is reduced, and the working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to an intelligent gait analysis auxiliary diagnosis system for knee joint function. Background Art

[0002] Gait analysis is an important medical detection method for disease monitoring and auxiliary diagnosis and treatment. It can assist in the diagnosis of various diseases, such as stroke, Parkinson's disease, cerebral palsy, etc., and predict the risk of falls, quality of life and disease mortality in the elderly. However, there are many problems with the current clinical gait analysis system, which seriously limits its widespread application in clinical practice.

[0003] The gait analysis systems currently widely used in clinical practice, such as the system of the British company Vicon, contain multiple high-precision infrared motion capture cameras. The system is expensive and requires professional medical staff to accurately paste reflective markers on the patient's body. The overall operation process is cumbersome and complicated, resulting in extremely high deployment and maintenance costs of the system, which cannot be promoted and applied in small and medium-sized hospitals.

[0004] The existing clinical gait analysis system has extremely high requirements for data acquisition and measurement accuracy, and the system relies on expensive precision equipment and complex operations. The data collection process is easily affected by environmental and human factors, resulting in insufficient measurement accuracy, which affects the accuracy of gait analysis.

[0005] Clinical gait analysis research lacks large-scale, multi-perspective clinical gait datasets, which limits the training and verification of gait analysis models. Without high-quality datasets as a foundation, it is difficult to make breakthrough progress in the research and application of gait analysis.

[0006] Gait analysis technology involves multiple disciplines such as computer vision, artificial intelligence, and medicine. The difficulties in technology integration and application promotion have led to the existing technology being relatively scattered and it is difficult to form a complete solution, which limits the application and popularization of gait analysis technology.

[0007] Therefore, those skilled in the art provide an intelligent gait analysis auxiliary diagnosis system for knee joint function to solve the problems raised in the above background technology. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides an intelligent gait analysis auxiliary diagnosis system for knee joint function to solve the problems in the background technology.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent gait analysis auxiliary diagnosis system for knee joint function, including a data acquisition module, a 3D human posture modeling module, a gait feature extraction module, a gait feature measurement and analysis module, a deep learning analysis and diagnosis module, an explanatory model and user interaction module, a system integration module, and a data management and storage module;

[0010] The data acquisition module collects human gait video data through a multi-view camera system; the 3D human posture modeling module converts the human key point data in the 2D image into 3D coordinates to construct a 3D human posture model; the gait feature extraction module extracts gait-related multi-dimensional features from the multi-view video data; the gait feature measurement and analysis module performs accurate quantitative measurement and analysis on the extracted gait features; the deep learning analysis and diagnosis module uses a deep learning model to analyze gait data to assist medical staff in making accurate diagnoses; the explanatory model and user interaction module provides the interpretability of the model reasoning process and evidence, and provides medical staff with tools to understand and verify the diagnosis results; the system integration module integrates each module to form a complete gait analysis auxiliary diagnosis system for functional verification; the data management and storage module manages and stores the collected gait data and analysis results.

[0011] Preferably, the data acquisition module includes a multi-view camera unit, a marker reference unit, and a time alignment unit; the multi-view camera unit can capture complete gait information at different viewing angles by shooting human gait videos at multiple angles; during the initial data acquisition, the marker reference unit pastes markers on key points of the human body under the guidance of medical experts to provide references for manual data annotation of key points; the time alignment unit performs time alignment on image frames shot by different cameras;

[0012] The 3D human posture modeling module includes a 2D key point calibration unit, a disparity measurement unit, and a 3D modeling unit; the 2D key point calibration unit accurately calibrates the key points of the human skeleton in the 2D image; the disparity measurement unit uses the disparity information of the multi-view images to calculate and reconstruct the 3D coordinates; the 3D modeling unit generates a 3D posture model of the human body based on the kernel surface geometry theory and the camera imaging principle.

[0013] Preferably, the gait feature extraction module includes a joint angle extraction unit, a gait parameter extraction unit, and a spatiotemporal feature extraction unit; the joint angle extraction unit extracts knee joint angle change data from a 3D posture model; the gait parameter extraction unit extracts gait feature parameters such as stride, step frequency, and walking speed; the spatiotemporal feature extraction unit extracts the spatiotemporal features of gait based on the spatiotemporal graph neural network technology to capture the dynamic changes of gait over time;

[0014] The gait feature measurement and analysis module includes a gait parameter measurement unit, a feature analysis unit, and a medical analysis unit; the gait parameter measurement unit performs accurate quantitative measurement on the extracted gait parameters; the feature analysis unit analyzes the gait feature data and provides a gait feature change report; the medical analysis unit combines clinical knowledge to analyze the medical significance of gait data to provide support for diagnosis.

[0015] Preferably, the deep learning analysis and diagnosis module includes a model training unit, a feature modeling unit, and a diagnosis reasoning unit; the model training unit uses a large-scale gait data set to train a deep learning model; the feature modeling unit models features from the gait data to identify abnormal gait patterns; the diagnosis reasoning unit performs diagnostic analysis on new gait data based on the trained model;

[0016] The explanatory model and user interaction module includes a reasoning and interpretation unit, a user interaction interface unit, and a feedback and adjustment unit; the reasoning and interpretation unit provides the reasoning process and basis of the diagnostic results to enhance the interpretability of the model; the user interaction interface unit designs a friendly user interface to display gait analysis results and explanations and support user queries and interactions; the feedback and adjustment unit receives user feedback and adjusts the model and analysis strategy.

[0017] Preferably, the system integration module includes an interface integration unit, a system interface unit, and a performance verification unit; the interface integration unit integrates various functional modules to ensure seamless connection of data and functions; the system interface unit constructs a complete system interface to support data collection, processing, analysis and result display; the performance verification unit performs overall performance testing and verification on the system;

[0018] The data management and storage module includes a data storage unit, a data retrieval unit, and a data security unit; the data storage unit securely stores gait videos, 3D posture models, gait feature data, and diagnostic results; the data retrieval unit provides efficient data retrieval and query functions to support users' quick access; the data security unit ensures the security and privacy of data to prevent data leakage and unauthorized access.

[0019] Preferably, the task of the 2D key point calibration unit is to identify and calibrate the key point coordinates of the human body from the image, and the algorithm includes a key point detection method based on a convolutional neural network, and its main formulas and steps are as follows:

[0020] Input image I, extract features through convolutional neural network and generate key point heat map H. The heat map corresponding to each key point is: H i =CNN(I), where H i is the heat map of the i-th key point;

[0021] The key point coordinates (x i ,y i ):

[0022] The formula is expressed in the heat map H i The peak position, i.e. the coordinates of the key point.

[0023] Preferably, the task of the disparity measurement unit is to calculate the disparity of key points based on multi-view images to perform 3D reconstruction, and its main formula and steps are as follows:

[0024] The basic matrix F is used to describe the geometric relationship between perspectives:

[0025] Where p1 = [x1, y1, 1] T , p2=[x2,y2,1] T is the coordinate of the corresponding point under the viewing angle;

[0026] The parallax calculation is based on the epipolar constraint. In the perspective image, the corresponding point of a point must be located on the epipolar line: l2 = Fp1, l1 = F T p2, where l2 and l1 are the epipolar lines of points p1 and p2.

[0027] Preferably, the task of the gait parameter extraction unit is to extract gait-related parameters from the 3D posture data. The stride is the distance between adjacent same feet touching the ground. Assuming that the 3D coordinates corresponding to the moments t1 and t2 of the left foot touching the ground are L1 (XL1, YL1, ZL1) and L2 (XL2, YL2, ZL2), the stride calculation formula is:

[0028] Stride = ∥L2-L1∥,

[0029]

[0030] Cadence is the number of steps completed per unit time. Assuming that N steps are completed within time T, the cadence calculation formula is: Cadence = N / T*60.

[0031] Preferably, the model training unit is based on a loss function, assuming that the prediction result of the model is The true label is y, and the loss function L can use mean square error or cross entropy loss;

[0032] Mean Square Error:

[0033] Cross Entropy Loss:

[0034] Preferably, the task of the diagnostic reasoning unit is to use the trained model to perform diagnostic analysis on new gait data and predict the results;

[0035] Assuming the model input is X, the weight is W and the bias is b, the calculation formula of the output layer is: z = WX + b, where z is the unactivated output;

[0036] Perform soft maximum activation on the output to obtain the category probability distribution p:

[0037] Among them, p i is the probability of the i-th class;

[0038] According to the probability distribution p, the category with the highest probability is taken as the diagnosis result:

[0039] The present invention provides an intelligent gait analysis auxiliary diagnosis system for knee joint function, which has the following beneficial effects:

[0040] 1. The present invention adopts non-contact, multi-angle camera technology and artificial intelligence algorithm to reduce dependence on expensive equipment and professional operations, reduce system deployment and maintenance costs, so that small and medium-sized hospitals can provide medical services related to gait analysis. In addition, through non-contact data collection, the step of sticking marking points on the patient is eliminated, the operation process is simplified, the dependence on professional medical staff is reduced, and work efficiency is improved.

[0041] 2. The present invention uses computer vision and deep learning technology to accurately calibrate key points of the human body at the sub-pixel level in 2D images, reconstruct 3D coordinates, ensure accurate measurement of gait parameters, and automatically extract and analyze gait features through a deep learning model to provide accurate and objective diagnostic results and improve diagnostic accuracy.

[0042] 3. The present invention combines the knowledge of medical experts to design an explainable deep learning model, provides the reasoning process and evidence of the diagnosis results, enhances the transparency and auditability of the system, enables medical staff to understand and verify the diagnosis results, improves the credibility of the system, and takes the lead in building a large-scale clinical gait dataset that includes cross-perspective human key point annotations, disease labels and expert interpretations, providing high-quality data support for gait analysis research and promoting research in the field of gait analysis.

[0043] 4. The present invention integrates multidisciplinary technologies such as computer vision, artificial intelligence and clinical medicine to form a complete gait analysis solution, promote technology integration and application, and improve the popularization and application level of gait analysis technology. It also reveals the key links and influencing factors of gait abnormalities through biomechanics and kinematics, guides rehabilitation evaluation and treatment, assists clinical diagnosis and efficacy evaluation, and provides automatically generated gait analysis reports to help medical staff diagnose and formulate treatment plans faster and more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural diagram of the present invention;

[0045] Figure 2 It is a schematic diagram of a data acquisition module of the present invention;

[0046] Figure 3 It is a schematic diagram of a 3D human body posture modeling module of the present invention;

[0047] Figure 4 Schematic diagram of a gait feature extraction module of the present invention;

[0048] Figure 5 It is a schematic diagram of a gait feature measurement and analysis module of the present invention;

[0049] Figure 6 A schematic diagram of a deep learning analysis and diagnosis module of the present invention;

[0050] Figure 7 It is a schematic diagram of the explanatory model and user interaction module of the present invention;

[0051] Figure 8 It is a schematic diagram of the system integration module of the present invention;

[0052] Fig. 9 It is a schematic diagram of the data management and storage module of the present invention;

[0053] Fig.10 Schematic diagram of the data acquisition module.

[0054] Fig.11 This is a schematic diagram of the collection results of the data collection module. DETAILED DESCRIPTION

[0055] In order to make the technical personnel in the technical field understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a partial embodiment of the present invention, not a complete embodiment. Based on the embodiment of the present invention, other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0056] Knee flexion angle

[0057] The knee joint moves forward and backward around the horizontal axis in the sagittal plane. The backward movement is flexion and the forward movement is extension. The knee joint angle in the upright prone position is defined as 0, the backward movement is positive, and the forward movement is negative.

[0058] Knee adduction angle

[0059] The movement of the knee joint around the sagittal axis in the frontal plane is adduction to the inside of the body and abduction to the outside of the body. When standing upright, the adduction and abduction angles of the knee joint are 0, adduction is positive and abduction is negative. The adduction and abduction angles of the knee joint are important indicators for judging X-shaped legs and O-shaped legs.

[0060] Knee angle

[0061] The knee angle is the angle formed by the hip-knee-ankle.

[0062] The present invention is described in detail below in conjunction with the accompanying drawings:

[0063] Example:

[0064] Please see attached Figure 1 - Attachment Fig.11 , an embodiment of the present invention provides an intelligent gait analysis auxiliary diagnosis system for knee joint function, including a data acquisition module, a 3D human posture modeling module, a gait feature extraction module, a gait feature measurement and analysis module, a deep learning analysis and diagnosis module, an explanatory model and user interaction module, a system integration module, and a data management and storage module;

[0065] The data acquisition module collects human gait video data through a multi-view camera system; the 3D human posture modeling module converts the human key point data in the 2D image into 3D coordinates to construct a 3D human posture model; the gait feature extraction module extracts gait-related multi-dimensional features from multi-view video data; the gait feature measurement and analysis module performs accurate quantitative measurement and analysis on the extracted gait features; the deep learning analysis and diagnosis module uses a deep learning model to analyze gait data to assist medical staff in making accurate diagnoses; the explanatory model and user interaction module provides the interpretability of the model reasoning process and evidence, and provides medical staff with tools to understand and verify the diagnosis results; the system integration module integrates various modules to form a complete gait analysis auxiliary diagnosis system for functional verification; the data management and storage module manages and stores the collected gait data and analysis results.

[0066] The benefits of the data acquisition module are that there is no need to stick markers on the patient, which simplifies the operation process and improves the patient's comfort. It also obtains more comprehensive gait data through multi-view cameras, improving the integrity and accuracy of the data. The benefits of the 3D human posture modeling module are that computer vision and deep learning technology are used to achieve high-precision 3D human posture modeling, reducing dependence on manual annotation and improving data processing efficiency. The benefits of the gait feature extraction module are that a variety of gait features including joint angles, stride length, and step frequency are extracted to provide richer analysis data. Graph neural network and recurrent neural network technology are used to extract the spatiotemporal characteristics of gait and improve the accuracy of feature extraction. The benefits of the gait feature measurement and analysis module are that the gait parameters are accurately quantified to provide high-precision analysis data, and combined with clinical knowledge , analyze the medical significance of gait data; the benefits of deep learning analysis and diagnosis module are to automatically analyze gait data through deep learning models, provide highly accurate diagnosis results, provide scientific diagnosis suggestions for medical staff, and reduce the risk of misdiagnosis; the benefits of explanatory model and user interaction module are to provide the reasoning process and basis of diagnosis results through explanatory models, improve the transparency and credibility of diagnosis results, provide a friendly user interaction interface, and facilitate medical staff to query and understand the analysis results; the benefits of system integration module are that each functional module is developed independently, which is convenient for system maintenance and upgrades, and the overall performance and stability of the system are ensured through integrated testing; the benefits of data management and storage module are to ensure data security and privacy, prevent data leakage, provide efficient data retrieval and query functions, and support fast access to data.

[0067] The data acquisition module includes a multi-view camera unit, a marker point reference unit, and a time alignment unit; the multi-view camera unit can capture complete gait information at different viewing angles by shooting human gait videos from multiple angles; during the initial data collection, the marker point reference unit pastes markers on key points of the human body under the guidance of medical experts to provide a reference for manual data annotation of key points; the time alignment unit performs time alignment on image frames shot by different cameras; the 3D human posture modeling module includes a 2D key point calibration unit, a disparity measurement unit, and a 3D modeling unit; the 2D key point calibration unit accurately calibrates the key points of the human skeleton in the 2D image; the disparity measurement unit uses the disparity information of multi-view images to calculate and reconstruct 3D coordinates; the 3D modeling unit generates a 3D posture model of the human body based on plane geometry theory and camera imaging principles; the gait feature extraction module includes a joint angle extraction unit, a gait parameter extraction unit, and a spatiotemporal feature extraction unit.

[0068] The joint angle extraction unit extracts the knee joint angle change data from the 3D posture model, including flexion / extension angle, internal rotation / external rotation angle, and varus / valgus angle.

[0069] First, we need to define three basic planes and their corresponding coordinate systems:

[0070] Sagittal plane (front-back direction): X axis (front-back), Y axis (up-down), Z axis (left-right)

[0071] Coronal plane (left and right direction): X axis (left and right), Y axis (up and down), Z axis (front and back)

[0072] Horizontal plane (up and down direction): X axis (front and back), Y axis (left and right), Z axis (up and down)

[0073] The calculation formula of the flexion / extension angle (sagittal plane) is:

[0074]

[0075] The calculation formula of the internal rotation / external rotation angle (horizontal plane) is:

[0076] θ rotation =arctan2(z cross ,x cross )

[0077] The calculation formula of the varus / valgus angle (coronal plane) is:

[0078]

[0079] Among them, P1 (x1, y1, z1) is the distal end of the femur, P2 (x2, y2, z2) is the proximal end of the tibia, and P0 (x0, y0, z0) is the reference point used to determine the rotation axis. cross 、z cross is the vector representing the effect of the rotation obtained by the cross product.

[0080] The gait parameter extraction unit extracts gait characteristic parameters such as stride, cadence, and walking speed; the spatiotemporal feature extraction unit extracts the spatiotemporal features of gait based on the spatiotemporal graph neural network technology, and captures the dynamic changes of gait over time; the gait feature measurement and analysis module includes a gait parameter measurement unit, a feature analysis unit, and a medical analysis unit; the gait parameter measurement unit performs accurate quantitative measurement of the extracted gait parameters; the feature analysis unit analyzes the data of gait characteristics and provides a gait feature change report; the medical analysis unit analyzes the medical significance of gait data in combination with clinical knowledge to provide support for diagnosis; the deep learning analysis and diagnosis module includes a model training unit, a feature modeling unit, and a diagnostic reasoning unit; the model training unit uses a large-scale gait data set to train a deep learning model; the feature modeling unit models features from the gait data to identify abnormal gait patterns; the diagnostic reasoning unit performs diagnostic analysis on new gait data based on the trained model; the explanatory model and user interaction module includes a reasoning interpretation unit, a user interaction interface unit, a feedback and adjustment unit, and a diagnostic reasoning unit. The whole unit; the reasoning and interpretation unit provides the reasoning process and basis of the diagnostic results to enhance the interpretability of the model; the user interaction interface unit designs a friendly user interface to display the gait analysis results and explanations and support user queries and interactions; the feedback and adjustment unit receives user feedback and adjusts the model and analysis strategy; the system integration module includes the interface integration unit, the system interface unit and the performance verification unit; the interface integration unit integrates various functional modules to ensure seamless connection of data and functions; the system interface unit builds a complete system interface to support data collection, processing, analysis and result display; the performance verification unit performs overall performance testing and verification on the system; the data management and storage module includes the data storage unit, the data retrieval unit and the data security unit; the data storage unit securely stores gait videos, 3D posture models, gait feature data and diagnostic results; the data retrieval unit provides efficient data retrieval and query functions to support users' quick access; the data security unit ensures the security and privacy of data to prevent data leakage and unauthorized access.

[0081] The benefits of the multi-view camera unit are that it provides comprehensive gait data, reduces information loss, improves data quality, and increases data redundancy and reliability through multi-view coverage; the benefits of the marker reference unit are that it improves the accuracy of key point annotation, provides high-quality annotation data, and makes subsequent automated data processing more accurate; the benefits of the time alignment unit are that it ensures the synchronization of multi-view data and improves the accuracy of data fusion and analysis; the benefits of the 2D key point calibration unit are that it provides accurate 2D key point data, lays the foundation for 3D reconstruction, and improves the accuracy of 3D posture modeling; the benefits of the parallax measurement unit are that it uses parallax information for accurate 3D reconstruction, improves reconstruction accuracy, and reduces errors through multi-view data fusion; the benefits of the 3D modeling unit are that it provides high-precision The 3D posture model of the unit provides reliable basic data for gait analysis, reduces manual intervention, and improves modeling efficiency; the benefits of the joint angle extraction unit provide key joint angle data, provide important reference for gait analysis and diagnosis, and improve the accuracy of knee joint function analysis; the benefits of the gait parameter extraction unit provide comprehensive gait feature parameters, enrich gait analysis data, and help identify gait abnormalities and disease risks; the benefits of the spatiotemporal feature extraction unit provide dynamic change data of gait, improve the timeliness and accuracy of analysis, and help identify complex gait patterns and potential problems; the benefits of the gait parameter measurement unit provide high-precision gait parameter data, provide a reliable basis for analysis and diagnosis, reduce measurement errors, and improve the accuracy of analysis results; the benefits of the feature analysis unit provide high-precision gait parameter data, provide a reliable basis for analysis and diagnosis, reduce measurement errors, and improve the accuracy of analysis results; the benefits of the feature analysis unit provide high-precision gait parameter data, provide a reliable basis for analysis and diagnosis, reduce measurement errors, and improve the accuracy of analysis results. The benefits of the analysis unit are as follows: It provides detailed reports on changes in gait characteristics, helps medical staff understand the progression of the disease, and supports the formulation of personalized rehabilitation treatment plans; the benefits of the medical analysis unit are to provide medically significant analysis results, assist doctors in making accurate diagnoses, and improve the application value of gait analysis in clinical diagnosis; the benefits of the model training unit are to provide powerful model training capabilities, improve the generalization performance of the model, and use big data to improve the accuracy of the diagnostic model; the benefits of the feature modeling unit are to provide deep modeling capabilities of gait characteristics, identify abnormal gait patterns, and improve the ability to identify gait abnormalities and diseases; the benefits of the diagnostic reasoning unit are to provide efficient diagnostic reasoning capabilities, quickly give diagnostic results, reduce the diagnostic workload of doctors, and improve diagnostic efficiency; the benefits of the reasoning explanation unit are to provide High transparency and credibility of diagnostic results, increase trust between doctors and patients, provide detailed reasoning process and basis, and facilitate doctors to verify diagnostic results; User interaction interface unit benefits Provide convenient user interaction interface, improve user experience, support user query and interaction, and increase system practicality; Feedback and adjustment unit benefits Provide user feedback mechanism, continuously optimize models and analysis strategies, improve system adaptability and user satisfaction; Interface integration unit benefits Provide modular system architecture, facilitate system maintenance and upgrades, ensure seamless connection of various functional modules, and improve the overall performance of the system; System interface unit benefits Provide a comprehensive system interface, support the realization of various functions, facilitate user operation and data management, and improve the practicality of the system;Benefits of the performance verification unit: Provide comprehensive testing and verification of system performance, ensure system reliability and stability, discover and solve potential problems in the system, and improve system stability; Benefits of the data storage unit: Provide a secure data storage solution to ensure data integrity and security, support large-scale data storage and management, and meet clinical application requirements; Benefits of the data retrieval unit: Provide convenient data retrieval functions to support users to quickly access required data, improve data management efficiency, and facilitate data query and use; Benefits of the data security unit: Provide data security protection to ensure patient privacy and data security, prevent data leakage and unauthorized access, and improve system security. ;

[0082] The task of the 2D key point calibration unit is to identify and calibrate the key point coordinates of the human body from the image. The algorithm includes a key point detection method based on a convolutional neural network. Its main formulas and steps are as follows:

[0083] Input image I, extract features through convolutional neural network and generate key point heat map H. The heat map corresponding to each key point is: H i =CNN(I), where H i is the heat map of the i-th key point;

[0084] The key point coordinates (x i ,y i ):

[0085] The formula is expressed in the heat map H i The peak position, i.e. the coordinates of the key point.

[0086] Step 1: Input image I: The system obtains human gait image I;

[0087] Step 2: Convolutional neural network feature extraction: Input image I into the pre-trained convolutional neural network to extract feature maps;

[0088] Step 3: Generate key point heat map: Generate multiple key point heat map H based on feature map i , each key point corresponds to a heat map;

[0089] Step 4: Heat map peak detection: i In the figure, find the peak position and extract the coordinates of the key points of the human body (x i ,y i ).

[0090] The task of the disparity measurement unit is to calculate the disparity of key points based on multi-view images and perform 3D reconstruction. Its main formulas and steps are as follows:

[0091] The basic matrix F is used to describe the geometric relationship between perspectives:

[0092] Where p1 = [x1, y1, 1] T , p2=[x2,y2,1] T is the coordinate of the corresponding point under the viewing angle;

[0093] The parallax calculation is based on the epipolar constraint. In the perspective image, the corresponding point of a point must be located on the epipolar line: l2 = Fp1, l1 = F T p2, where l2 and l1 are the epipolar lines of points p1 and p2.

[0094] Step 1: Input multi-view images: obtain view images l1 and l2 of the same scene;

[0095] Step 2: Feature point detection and matching: Detect feature points in images l1 and l2;

[0096] {P1}=DetectFeatures(I1), {P2}=DetectFeatures(I2)

[0097] Step 3: Match feature point pairs; {(P1, P2)} = MatchFeatures({P1}, {P2})

[0098] Step 4: Calculate the basic matrix F: Calculate the basic matrix F using the matched feature point pairs:

[0099]

[0100] Step 5: Calculate the epipolar line: Calculate the epipolar line of the feature point in the image from another perspective.

[0101] l2=Fp1,l1=F T p2,

[0102] Step 6: Calculate the parallax: Calculate the parallax according to the epipolar constraint, d = ||P1-P2||.

[0103] The task of the gait parameter extraction unit is to extract gait-related parameters from the 3D posture data. The stride is the distance between the adjacent same feet touching the ground. Assuming that the 3D coordinates corresponding to the moments t1 and t2 of the left foot touching the ground are L1 (XL1, YL1, ZL1) and L2 (XL2, YL2, ZL2), the stride calculation formula is:

[0104] Stride = ∥L2-L1∥,

[0105]

[0106] Cadence is the number of steps completed per unit time. Assuming that N steps are completed within time T, the cadence calculation formula is: Cadence = N / T*60.

[0107] The model training unit is based on the loss function, assuming that the prediction result of the model is The true label is y, and the loss function L can use mean square error or cross entropy loss;

[0108] Mean Square Error:

[0109] Cross Entropy Loss:

[0110] The task of the diagnostic reasoning unit is to use the trained model to perform diagnostic analysis on new gait data and predict the results;

[0111] Assuming the model input is X, the weight is W and the bias is b, the calculation formula of the output layer is: z = WX + b, where z is the unactivated output;

[0112] Perform soft maximum activation on the output to obtain the category probability distribution p:

[0113] Among them, p i is the probability of the i-th class;

[0114] According to the probability distribution p, the category with the highest probability is taken as the diagnosis result:

[0115] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent gait analysis auxiliary diagnosis system for knee joint function, characterized in that: It includes data acquisition module, 3D human posture modeling module, gait feature extraction module, gait feature measurement and analysis module, deep learning analysis and diagnosis module, explanatory model and user interaction module, system integration module, data management and storage module; The data acquisition module collects human gait video data through a multi-view camera system; the 3D human posture modeling module converts the human key point data in the 2D image into 3D coordinates to construct a 3D human posture model; the gait feature extraction module extracts gait-related multi-dimensional features from the multi-view video data; the gait feature measurement and analysis module performs accurate quantitative measurement and analysis on the extracted gait features; the deep learning analysis and diagnosis module uses a deep learning model to analyze gait data to assist medical staff in making accurate diagnoses; the explanatory model and user interaction module provides the interpretability of the model reasoning process and evidence, and provides medical staff with tools to understand and verify the diagnosis results; the system integration module integrates each module to form a complete gait analysis auxiliary diagnosis system for functional verification; the data management and storage module manages and stores the collected gait data and analysis results.

2. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1 is characterized in that: The data acquisition module includes a multi-view camera unit, a marker reference unit, and a time alignment unit; the multi-view camera unit can capture complete gait information at different viewing angles by shooting human gait videos at multiple angles; During the initial data collection, the marker reference unit pastes markers on key points of the human body under the guidance of medical experts to provide references for manual data annotation of key points; the time alignment unit performs time alignment on image frames captured by different cameras; The 3D human posture modeling module includes a 2D key point calibration unit, a disparity measurement unit, and a 3D modeling unit; the 2D key point calibration unit accurately calibrates the key points of the human skeleton in the 2D image; the disparity measurement unit uses the disparity information of the multi-view images to calculate and reconstruct the 3D coordinates; the 3D modeling unit generates a 3D posture model of the human body based on the kernel surface geometry theory and the camera imaging principle.

3. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1, characterized in that: The gait feature extraction module includes a joint angle extraction unit, a gait parameter extraction unit, and a spatiotemporal feature extraction unit; the gait parameter extraction unit extracts gait feature parameters such as stride, step frequency, and walking speed; the spatiotemporal feature extraction unit extracts the spatiotemporal features of gait based on the spatiotemporal graph neural network technology to capture the dynamic changes of gait over time; The joint angle extraction unit extracts knee joint angle change data from the 3D posture model, including flexion / extension angle, internal rotation / external rotation angle, and varus / valgus angle. The calculation formula of the flexion / extension angle is: The calculation formula of the internal rotation / external rotation angle is: θ rotation =arctan2(z cross , x cross ) The calculation formula of the varus / valgus angle is: Among them, P1 (x1, y1, z1) is the distal end of the femur, P2 (x2, y2, z2) is the proximal end of the tibia, and P0 (x0, y0, z0) is the reference point used to determine the rotation axis. cross 、z cross is the vector representing the rotation effect obtained by the cross product; The gait feature measurement and analysis module includes a gait parameter measurement unit, a feature analysis unit, and a medical analysis unit; the gait parameter measurement unit performs accurate quantitative measurement on the extracted gait parameters; the feature analysis unit analyzes the gait feature data and provides a gait feature change report; the medical analysis unit combines clinical knowledge to analyze the medical significance of gait data to provide support for diagnosis.

4. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1, characterized in that: The deep learning analysis and diagnosis module includes a model training unit, a feature modeling unit, and a diagnosis reasoning unit; the model training unit uses a large-scale gait data set to train a deep learning model; the feature modeling unit models features from gait data to identify abnormal gait patterns; the diagnosis reasoning unit performs diagnostic analysis on new gait data based on the trained model; The explanatory model and user interaction module includes a reasoning explanation unit, a user interaction interface unit, and a feedback and adjustment unit; The reasoning and explanation unit provides the reasoning process and basis of the diagnosis results, enhancing the interpretability of the model; The user interaction interface unit designs a user-friendly user interface to display gait analysis results and explanations and support user inquiries and interactions; the feedback and adjustment unit receives user feedback and adjusts the model and analysis strategy.

5. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1, characterized in that: The system integration module includes an interface integration unit, a system interface unit, and a performance verification unit; the interface integration unit integrates various functional modules to ensure seamless connection of data and functions; the system interface unit constructs a complete system interface to support data collection, processing, analysis and result display; the performance verification unit performs overall performance testing and verification on the system; The data management and storage module includes a data storage unit, a data retrieval unit, and a data security unit; the data storage unit securely stores gait videos, 3D posture models, gait feature data, and diagnostic results; the data retrieval unit provides efficient data retrieval and query functions to support users' quick access; the data security unit ensures the security and privacy of data to prevent data leakage and unauthorized access.

6. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1, characterized in that: The task of the 2D key point calibration unit is to identify and calibrate the key point coordinates of the human body from the image. The algorithm includes a key point detection method based on a convolutional neural network, and its main formulas and steps are as follows: Input image I, extract features through convolutional neural network and generate key point heat map H. The heat map corresponding to each key point is: H i =CNN(I), where H i is the heat map of the i-th key point; The key point coordinates (x i ,y i ): The formula is expressed in the heat map H i The peak position, i.e. the coordinates of the key point.

7. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1, characterized in that: The task of the disparity measurement unit is to calculate the disparity of key points based on multi-view images and perform 3D reconstruction. The main formula and steps are as follows: The basic matrix F is used to describe the geometric relationship between perspectives: Where p1 = [ x1,y1,1] T , p2=[x2,y2,1] T is the coordinate of the corresponding point under the viewing angle; The parallax calculation is based on the epipolar constraint. In the perspective image, the corresponding point of a point must be located on the epipolar line: l2 = Fp1, l1 = F T p2, where l2 and l1 are the epipolar lines of points p1 and p2.

8. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1, characterized in that: The task of the gait parameter extraction unit is to extract gait-related parameters from the 3D posture data. The stride is the distance between adjacent same feet touching the ground. Assuming that the 3D coordinates corresponding to the moments t1 and t2 of the left foot touching the ground are L1 (XL1, YL1, ZL1) and L2 (XL2, YL2, ZL2), the stride calculation formula is: Stride = ∥L2-L1∥, Cadence is the number of steps completed per unit time. Assuming that N steps are completed within time T, the cadence calculation formula is: Cadence = N / T*60.

9. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1, characterized in that: The model training unit is based on the loss function, assuming that the prediction result of the model is The true label is y, and the loss function L can use mean square error or cross entropy loss; Mean Square Error: Cross Entropy Loss:

10. The intelligent gait analysis auxiliary diagnosis system for knee joint function according to claim 1, characterized in that: The task of the diagnostic reasoning unit is to use the trained model to perform diagnostic analysis on new gait data and predict the results; Assuming the model input is X, the weight is W and the bias is b, the calculation formula of the output layer is: z = WX + b, where z is the inactive output; Perform soft maximum activation on the output to obtain the category probability distribution p: Among them, p i is the probability of the i-th class; According to the probability distribution p, the category with the highest probability is taken as the diagnosis result:

Images