Intelligent gait analysis auxiliary diagnosis system for knee joint function

Abstract

The application relates to the medical technical field and discloses an intelligent gait analysis auxiliary diagnosis system for knee joint function, which comprises a data acquisition module, a 3D human posture modeling module, a gait feature extraction module, a gait feature measurement and analysis module, a deep learning analysis and diagnosis module, an explanatory model and user interaction module, a system integration module and a data management and storage module; the data acquisition module collects human gait video data through a multi-view camera system. The non-contact multi-view camera technology and the artificial intelligence algorithm are adopted to reduce the dependence on expensive equipment and professional operation, reduce the deployment and maintenance cost of the system, enable small and medium-sized hospitals to carry out medical services related to gait analysis, and through the non-contact data acquisition mode, the step of pasting mark points on the patient is omitted, the operation process is simplified, the dependence on professional medical staff is reduced, and the work efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the medical technology field, in particular to a kind of intelligent knee function gait analysis auxiliary diagnosis system. BACKGROUND

[0002] Gait analysis is an important medical detection means for disease monitoring and auxiliary diagnosis and treatment, which can assist in the diagnosis of various diseases such as stroke, Parkinson's disease, cerebral palsy, etc., and predict the risk of falling, quality of life and disease mortality of the elderly. However, the current clinical gait analysis system has many problems, which seriously limits its wide application in clinical practice.

[0003] The widely used gait analysis system in clinical practice, such as the system of British Vicon Company, contains multiple high-precision infrared light motion capture cameras. The system is expensive and requires professional medical staff to accurately paste reflective marker points on the patient's body. The overall operation process is complicated, resulting in high deployment and maintenance costs, which makes it difficult to be popularized and applied in small and medium-sized hospitals.

[0004] The existing clinical gait analysis system requires high precision for data acquisition and measurement, and the system relies on expensive precision equipment and complex operation. The data collection process is easily affected by environmental and human factors, resulting in insufficient measurement accuracy and affecting the accuracy of gait analysis.

[0005] Clinical gait analysis research lacks large-scale, multi-angle clinical gait dataset, which limits the training and verification of gait analysis model. Without high-quality dataset as the basis, the research and application of gait analysis are difficult to make breakthrough progress.

[0006] Gait analysis technology involves computer vision, artificial intelligence, medicine and other multidisciplinary fields, and the technology integration and application promotion are difficult, resulting in scattered existing technology, which is difficult to form a complete solution and limits the application and popularization of gait analysis technology.

[0007] Therefore, the skilled person in the art provides an intelligent knee function gait analysis auxiliary diagnosis system to solve the problems in the background art. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides an intelligent knee function gait analysis auxiliary diagnosis system to solve the problems in the background art.

[0009] To achieve the above purpose, the present application is realized by the following technical scheme: an intelligent knee function gait analysis auxiliary diagnosis system, comprising a data acquisition module, a 3D human posture modeling module, a gait feature extraction module, a gait feature measurement and analysis module, a deep learning analysis and diagnosis module, an explanatory model and user interaction module, a system integration module, a data management and storage module.

[0010] The data acquisition module collects human gait video data through a multi-view camera system; the 3D human posture modeling module converts human key point data in a 2D image into 3D coordinates to construct a 3D human posture model; the gait feature extraction module extracts multi-dimensional gait-related features from multi-view video data; the gait feature measurement and analysis module performs accurate quantitative measurement and analysis on the extracted gait features; the deep learning analysis and diagnosis module analyzes gait data using a deep learning model to assist medical personnel in accurate diagnosis; the explanatory model and user interaction module provides model reasoning process and evidence interpretability, providing a tool for medical personnel to understand and verify the diagnosis result; the system integration module integrates various modules to form a complete gait analysis auxiliary diagnosis system for function verification; and the data management and storage module manages and stores the collected gait data and analysis results.

[0011] Preferably, the data acquisition module includes a multi-view camera unit, a marker point reference unit, and a time alignment unit; the multi-view camera unit captures human gait video from multiple angles to capture complete gait information at different angles; the marker point reference unit, in initial data acquisition, pastes markers on human key points under the guidance of medical experts to provide a reference for manual data labeling of key points; and the time alignment unit performs time alignment on image frames captured by different cameras.

[0012] The 3D human posture modeling module includes a 2D key point calibration unit, a parallax measurement unit, and a 3D modeling unit; the 2D key point calibration unit accurately calibrates key points of human bones in a 2D image; the parallax measurement unit calculates reconstructed 3D coordinates using parallax information of multi-view images; and the 3D modeling unit generates a 3D posture model of the human body based on kernel surface geometry theory and camera imaging principles.

[0013] Preferably, the gait feature extraction module includes a joint angle extraction unit, a gait parameter extraction unit, and a spatio-temporal feature extraction unit; the joint angle extraction unit extracts knee joint angle change data from the 3D posture model; the gait parameter extraction unit extracts gait feature parameters such as stride, step frequency, and walking speed; and the spatio-temporal feature extraction unit extracts spatio-temporal features of gait based on spatio-temporal graph neural network technology to capture dynamic changes of gait over time.

[0014] The gait feature measurement and analysis module includes a gait parameter measurement unit, a feature analysis unit, and a medical analysis unit; the gait parameter measurement unit performs accurate quantitative measurement on the extracted gait parameters; the feature analysis unit analyzes gait feature data and provides a gait feature change report; and the medical analysis unit analyzes the medical significance of gait data in combination with clinical knowledge to provide support for diagnosis.

[0015] Preferably, the deep learning analysis and diagnosis module includes a model training unit, a feature modeling unit, and a diagnosis reasoning unit; the model training unit trains a deep learning model using a large-scale gait dataset; the feature modeling unit models features from gait data to identify abnormal gait patterns; and the diagnosis reasoning unit performs diagnostic analysis on new gait data based on the trained model.

[0016] The explanatory model and user interaction module includes a reasoning explanation unit, a user interaction interface unit, and a feedback and adjustment unit; the reasoning explanation unit provides reasoning process and basis for diagnostic results, enhancing the explainability of the model; the user interaction interface unit designs a friendly user interface to display gait analysis results and explanations, supporting user queries and interactions; and the feedback and adjustment unit receives user feedback to adjust the model and analysis strategy.

[0017] Preferably, the system integration module includes an interface integration unit, a system interface unit, and a performance verification unit; the interface integration unit integrates various functional modules to ensure seamless connection of data and functions; the system interface unit builds a complete system interface to support data collection, processing, analysis, and result display; and the performance verification unit performs overall performance testing and verification of the system.

[0018] The data management and storage module includes a data storage unit, a data retrieval unit, and a data security unit; the data storage unit securely stores gait video, 3D pose model, gait feature data, and diagnostic results; the data retrieval unit provides efficient data retrieval and query functions to support fast user access; and the data security unit ensures data security and privacy, preventing data leakage and unauthorized access.

[0019] Preferably, the task of the 2D key point labeling unit is to identify and label the key point coordinates of the human body from the image, and the algorithm includes a key point detection method based on a convolutional neural network, with the main formula and steps as follows:

[0020] Input image I, extract features through convolutional neural network and generate key point heat map H, each key point corresponds to a heat map: H i = CNN(I), where H i is the heat map of the i-th key point.

[0021] Extract key point coordinates (x i ,y i ) through the peak position of the heat map:

[0022] The formula represents the peak position in the heat map H i , i.e., the coordinates of the key point.

[0023] Preferably, the task of the parallax measurement unit is to calculate the parallax of the key points according to the multi-view images, and the main formula and steps for 3D reconstruction are as follows:

[0024] The fundamental matrix F is used to describe the geometric relationship between the views:

[0025] Where p1 = [x1, y1, 1] T , p2 = [x2, y2, 1] T are the corresponding point coordinates under the views;

[0026] The parallax calculation is based on the epipolar constraint, and in the view images, the corresponding points of a certain point must be located on the epipolar line: l2 = Fp1, l1 = F T p2, where l2 and l1 are the epipolar lines of points p1 and p2.

[0027] Preferably, the task of the gait parameter extraction unit is to extract the parameters related to gait from the 3D pose data, and the step length is the distance between the ground contacts of adjacent same feet. Assuming that the 3D coordinates corresponding to the ground contacts of the left foot at times t1 and t2 are L1 (XL1, YL1, ZL1) and L2 (XL2, YL2, ZL2), the step length calculation formula is:

[0028] Step length = ||L2 - L1||,

[0029]

[0030] The step frequency is the number of steps completed per unit time, and assuming that N steps are completed in time T, the step frequency calculation formula is: step frequency = N / T*60.

[0031] Preferably, the model training unit is based on a loss function, and assuming that the prediction result of the model is The true label is y, and the loss function L can adopt mean square error or cross-entropy loss;

[0032] Mean square error:

[0033] Cross-entropy loss:

[0034] Preferably, the task of the diagnosis reasoning unit is to use the trained model to perform diagnostic analysis on new gait data, and the prediction result;

[0035] Assuming that the input of the model is X, the weight is W and the bias is b, the calculation formula of the output layer is: z = WX + b, where z is the unactivated output;

[0036] Soft maximum activation is performed on the output to obtain the class probability distribution p:

[0037] wherein p i is the probability of the i-th class;

[0038] According to the probability distribution p, the class with the highest probability is taken as the diagnosis result:

[0039] The application provides a gait analysis auxiliary diagnosis system for intelligent knee joint function. The system has the following advantages:

[0040] 1. The application uses non-contact, multi-view camera technology and artificial intelligence algorithms to reduce dependence on expensive equipment and professional operation, reduce system deployment and maintenance costs, and enable small and medium-sized hospitals to provide gait analysis-related medical services. Through the non-contact data acquisition method, the step of pasting marker points on the patient is eliminated, the operation process is simplified, the dependence on professional medical personnel is reduced, and the work efficiency is improved.

[0041] 2. The application uses computer vision and deep learning technology to accurately calibrate sub-pixel level human key points in 2D images, reconstruct 3D coordinates, ensure accurate measurement of gait parameters, and automatically extract and analyze gait features through a deep learning model to provide accurate and objective diagnosis results and improve diagnosis accuracy.

[0042] 3. The application combines medical expert knowledge to design an interpretable deep learning model, provides a reasoning process and evidence for the diagnosis result, enhances the transparency and auditability of the system, enables medical personnel to understand and verify the diagnosis result, improves the credibility of the system, and first constructs a large-scale clinical gait dataset containing cross-view human key point annotation, disease label and expert explanation, provides high-quality data support for gait analysis research, and promotes research in the field of gait analysis.

[0043] 4. The application integrates computer vision, artificial intelligence and clinical medicine and other multidisciplinary technologies to form a complete gait analysis solution, promotes technology integration and application, improves the popularization and application level of gait analysis technology, and reveals the key links and influencing factors of gait abnormalities through biomechanics and kinematics, guides rehabilitation evaluation and treatment, assists clinical diagnosis and efficacy evaluation, provides automatically generated gait analysis reports, and helps medical personnel to diagnose and develop treatment plans more quickly and accurately. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a structural diagram of the application;

[0045] Figure 2 is a schematic diagram of the data acquisition module of the application;

[0046] Figure 3 is a schematic diagram of the 3D human posture modeling module of the application;

[0047] Figure 4 Gait feature extraction module schematic diagram of the present application;

[0048] Figure 5 Gait feature measurement and analysis module schematic diagram of the present application;

[0049] Figure 6 Deep learning analysis and diagnosis module schematic diagram of the present application;

[0050] Figure 7 Explanatory model and user interaction module schematic diagram of the present application;

[0051] Figure 8 System integration module schematic diagram of the present application;

[0052] Figure 9 Data management and storage module schematic diagram of the present application;

[0053] Figure 10 Data acquisition module acquisition schematic diagram.

[0054] Figure 11 Data acquisition module acquisition result schematic diagram. DETAILED DESCRIPTION

[0055] In order for those skilled in the art to understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0056] Knee flexion angle

[0057] The knee joint moves forward and backward in the sagittal plane around the transverse axis, backward for flexion and forward for extension. The knee joint angle is defined as 0 in the upright gait, positive for backward flexion and negative for forward extension.

[0058] Knee adduction-abduction angle

[0059] The knee joint moves in the frontal plane around the sagittal axis, inward for adduction and outward for abduction. The knee joint adduction-abduction angle is defined as 0 in the upright gait, positive for adduction and negative for abduction. The knee joint adduction-abduction angle is an important indicator for judging X-shaped legs and O-shaped legs.

[0060] Knee angle

[0061] The knee angle is the angle formed by the hip-knee-ankle.

[0062] The application will be described in detail below with reference to the drawings:

[0063] Embodiment:

[0064] Please refer to the drawings Figure 1 - the drawings Figure 11 The embodiment of the application provides a gait analysis auxiliary diagnosis system for intelligent knee joint function, which comprises a data acquisition module, a 3D human posture modeling module, a gait feature extraction module, a gait feature measurement and analysis module, a deep learning analysis and diagnosis module, an explanatory model and user interaction module, a system integration module and a data management and storage module.

[0065] The data acquisition module acquires human gait video data through a multi-view camera system; the 3D human posture modeling module converts human key point data in a 2D image into 3D coordinates to construct a 3D human posture model; the gait feature extraction module extracts multi-dimensional gait-related features from multi-view video data; the gait feature measurement and analysis module accurately quantitatively measures and analyzes the extracted gait features; the deep learning analysis and diagnosis module analyzes gait data by using a deep learning model to assist medical personnel in accurate diagnosis; the explanatory model and user interaction module provides model reasoning process and evidence interpretability to provide a tool for medical personnel to understand and verify diagnosis results; the system integration module integrates all modules to form a complete gait analysis auxiliary diagnosis system for function verification; and the data management and storage module manages and stores acquired gait data and analysis results.

[0066] The data acquisition module has the advantages of not needing to paste marker points on the patient's body, simplifying the operation process, improving the comfort of the patient, and obtaining more comprehensive gait data through multiple view cameras to improve the integrity and accuracy of the data; the 3D human posture modeling module has the advantages of utilizing computer vision and deep learning technology to realize high-precision 3D human posture modeling, reducing the dependence on manual annotation, and improving the data processing efficiency; the gait feature extraction module has the advantages of extracting various gait features including joint angle, stride, and step frequency, providing more abundant analysis data, utilizing graph neural network and recurrent neural network technology to extract the spatio-temporal features of gait, and improving the accuracy of feature extraction; the gait feature measurement and analysis module has the advantages of accurately quantifying the gait parameters, providing high-precision analysis data, and analyzing the medical significance of the gait data in combination with clinical knowledge; the deep learning analysis and diagnosis module has the advantages of automatically analyzing the gait data through a deep learning model, providing high-accuracy diagnosis results, and providing scientific diagnosis suggestions for medical staff to reduce the risk of misdiagnosis; the interpretive model and user interaction module has the advantages of providing the reasoning process and basis of the diagnosis results through the interpretive model, improving the transparency and credibility of the diagnosis results, and providing a friendly user interaction interface for medical staff to query and understand the analysis results; the system integration module has the advantages of independent development of each functional module for easy maintenance and upgrading of the system, and through integration testing to ensure the overall performance and stability of the system; the data management and storage module has the advantages of ensuring the security and privacy of the data, preventing data leakage, providing efficient data retrieval and query functions, and supporting fast data access.

[0067] The data acquisition module includes a multi-view camera unit, a marker point reference unit, and a time alignment unit; the multi-view camera unit captures human gait videos from multiple angles, capturing complete gait information from different perspectives; the marker point reference unit, in the initial data acquisition, uses medical experts to guide the pasting of markers on key points of the human body, providing a reference for manual data annotation of key points; the time alignment unit aligns the image frames captured by different cameras in time; the 3D human posture modeling module includes a 2D key point calibration unit, a parallax measurement unit, and a 3D modeling unit; the 2D key point calibration unit accurately calibrates the key points of the human skeleton in the 2D image; the parallax measurement unit calculates the reconstructed 3D coordinates using the parallax information of multi-view images; the 3D modeling unit generates a 3D posture model of the human body based on the theory of plane geometry and the principle of camera imaging; the gait feature extraction module includes a joint angle extraction unit, a gait parameter extraction unit, and a spatio-temporal feature extraction unit.

[0068] The joint angle extraction unit extracts knee joint angle change data from the 3D posture model; including flexion / extension angle, internal / external rotation angle, and inversion / external rotation angle.

[0069] First, three basic planes and corresponding coordinate systems need to be defined:

[0070] Sagittal plane (front-back direction): X-axis (front-back), Y-axis (up-down), Z-axis (left-right)

[0071] Coronal plane (left-right direction): X-axis (left-right), Y-axis (up-down), Z-axis (front-back)

[0072] Horizontal plane (up-down direction): X-axis (front-back), Y-axis (left-right), Z-axis (up-down)

[0073] The calculation formula of the flexion / extension angle (sagittal plane) is:

[0074]

[0075] The calculation formula of the internal / external rotation angle (horizontal plane) is:

[0076] θ rotation = arctan2(z cross , x cross )

[0077] The calculation formula of the internal / external rotation angle (coronal plane) is:

[0078]

[0079] Where P1 (x1, y1, z1) is the distal femoral point, P2 (x2, y2, z2) is the proximal tibial point, P0 (x0, y0, z0) is the reference point for determining the rotation axis, x cross , z cross is the vector representing the rotation effect obtained by the cross product.

[0080] The gait parameter extraction unit extracts gait characteristic parameters such as stride length, stride frequency, and walking speed; the spatio-temporal feature extraction unit extracts spatio-temporal features of gait based on spatio-temporal graph neural network technology, capturing dynamic changes of gait over time; the gait feature measurement and analysis module includes a gait parameter measurement unit, a feature analysis unit, and a medical analysis unit; the gait parameter measurement unit accurately quantitatively measures the extracted gait parameters; the feature analysis unit analyzes gait feature data and provides a gait feature change report; the medical analysis unit analyzes the medical significance of gait data in combination with clinical knowledge to provide support for diagnosis; the deep learning analysis and diagnosis module includes a model training unit, a feature modeling unit, and a diagnosis reasoning unit; the model training unit trains deep learning models using large-scale gait data sets; the feature modeling unit models features from gait data to identify abnormal gait patterns; the diagnosis reasoning unit diagnoses and analyzes new gait data based on the trained model; the explanatory model and user interaction module includes a reasoning explanation unit, a user interaction interface unit, and a feedback and adjustment unit; the reasoning explanation unit provides reasoning process and basis for diagnosis results, enhancing the explainability of the model; the user interaction interface unit designs a friendly user interface to display gait analysis results and explanations, supporting user queries and interactions; the feedback and adjustment unit receives user feedback to adjust the model and analysis strategy; the system integration module includes an interface integration unit, a system interface unit, and a performance verification unit; the interface integration unit integrates various functional modules to ensure seamless connection of data and functions; the system interface unit builds a complete system interface to support data collection, processing, analysis, and result display; the performance verification unit performs overall performance testing and verification on the system; the data management and storage module includes a data storage unit, a data retrieval unit, and a data security unit; the data storage unit securely stores gait video, 3D pose models, gait feature data, and diagnosis results; the data retrieval unit provides efficient data retrieval and query functions to support fast user access; the data security unit ensures data security and privacy to prevent data leakage and unauthorized access.

[0081] The multi-view camera unit provides comprehensive gait data, reduces information loss, improves data quality, and increases data redundancy and reliability through multi-view coverage. The marker point reference unit improves the accuracy of key point labeling, provides high-quality labeled data, and makes subsequent automated data processing more accurate. The time alignment unit ensures the synchronization of multi-view data, improves the accuracy of data fusion and analysis, and provides accurate 2D key point data for 3D reconstruction, improving the accuracy of 3D pose modeling. The parallax measurement unit uses parallax information for accurate 3D reconstruction, improves reconstruction accuracy, and reduces errors through multi-view data fusion. The 3D modeling unit provides high-precision 3D pose models, provides reliable basic data for gait analysis, reduces manual intervention, and improves modeling efficiency. The joint angle extraction unit provides key joint angle data, provides important references for gait analysis and diagnosis, and improves the accuracy of knee joint function analysis. The gait parameter extraction unit provides comprehensive gait feature parameters, enriches gait analysis data, and helps identify gait abnormalities and disease risks. The spatio-temporal feature extraction unit provides dynamic gait change data, improves the timeliness and accuracy of analysis, and helps identify complex gait patterns and potential problems. The gait parameter measurement unit provides high-precision gait parameter data, provides reliable basis for analysis and diagnosis, reduces measurement errors, and improves the accuracy of analysis results. The feature analysis unit provides detailed gait feature change reports, helps medical staff understand disease progression, and supports the development of personalized rehabilitation treatment plans. The medical analysis unit provides medical significance analysis results, assists doctors in making accurate diagnoses, and improves the application value of gait analysis in clinical diagnosis. The model training unit provides powerful model training capabilities, improves the generalization performance of the model, and improves the accuracy of the diagnosis model using big data. The feature modeling unit provides deep modeling capabilities for gait features, identifies abnormal gait patterns, and improves the ability to identify gait abnormalities and diseases. The diagnosis reasoning unit provides efficient diagnosis reasoning capabilities, quickly provides diagnosis results, reduces the workload of doctors, and improves diagnosis efficiency. The reasoning interpretation unit improves the transparency and credibility of diagnosis results, increases the trust of doctors and patients, provides detailed reasoning process and basis, and facilitates doctors to verify diagnosis results. The user interaction interface unit provides a convenient user interaction interface, improves user experience, supports user query and interaction, and increases the practicality of the system. The feedback and adjustment unit provides a user feedback mechanism, continuously optimizes the model and analysis strategy, and improves the adaptability and user satisfaction of the system. The interface integration unit provides a modular system architecture, facilitates system maintenance and upgrading, ensures seamless connection of each functional module, and improves the overall performance of the system. The system interface unit provides a comprehensive system interface, supports the implementation of various functions, facilitates user operation and data management, and improves the practicality of the system.The performance verification unit provides comprehensive testing and verification of system performance, ensures the reliability and stability of the system, finds and solves potential problems in the system, and improves the stability of the system; The data storage unit provides a secure data storage solution, ensures the integrity and security of the data, supports the storage and management of large-scale data, and meets the needs of clinical applications; The data retrieval unit provides convenient data retrieval functions, supports users to quickly access the required data, improves data management efficiency, and facilitates data query and use; The data security unit provides data security protection, ensures patient privacy and data security, prevents data leakage and unauthorized access, and improves system security.

[0082] The task of the 2D key point calibration unit is to identify and calibrate the key point coordinates of the human body from the image. The algorithm includes a key point detection method based on a convolutional neural network, and its main formulas and steps are as follows:

[0083] Input image I, extract features and generate key point heat map H through convolutional neural network, and each key point corresponds to a heat map: H i = CNN(I), where H i is the heat map of the i-th key point.

[0084] Extract key point coordinates (x i ,y i ) through the peak position of the heat map:

[0085] The formula represents the peak position in the heat map H i , i.e. the coordinates of the key point.

[0086] Step 1, input image I: the system acquires human gait image I;

[0087] Step 2, convolutional neural network feature extraction: input image I into a pre-trained convolutional neural network to extract feature maps;

[0088] Step 3, generate key point heat map: generate multi-key point heat map H i according to the feature map, and each key point corresponds to a heat map;

[0089] Step 4, heat map peak detection: find the peak position in each heat map H i , and extract the coordinates (x i ,y i ) of the key points of the human body.

[0090] The task of the disparity measurement unit is to calculate the disparity of the key points according to the multi-view images, and to perform 3D reconstruction, and its main formulas and steps are as follows:

[0091] The fundamental matrix F is used to describe the geometric relationship between the perspectives:

[0092] where p1=[x1,y1,1] T p2=[x2,y2,1] T are the corresponding point coordinates in the view;

[0093] The disparity is calculated based on the epipolar constraint, in the view images, the corresponding point of a certain point must be located on the epipolar line: l2=Fp1, l1=F T p2, where l2 and l1 are the epipolar lines of points p1 and p2.

[0094] The first step, input multi-view images: obtain the view images l1 and l2 of the same scene;

[0095] The second step, feature point detection and matching: detect feature points in images l1 and l2;

[0096] {P1}=DetectFeatures(I1), {P2}=DetectFeatures(I2)

[0097] The third step, matching feature points; {(P1, P2)}=MatchFeatures({P1}, {P2})

[0098] The fourth step, calculating the fundamental matrix F: using the matched feature point pairs to calculate the fundamental matrix F:

[0099]

[0100] The fifth step, calculating the epipolar line: calculating the epipolar line of the feature points in the other view image,

[0101] l2=Fp1, l1=F T p2,

[0102] The sixth step, calculating the disparity: calculating the disparity according to the epipolar constraint, d=||P1-P2||.

[0103] The task of the gait parameter extraction unit is to extract the parameters related to the gait from the 3D pose data. The step length is the distance between the ground contacts of the adjacent same foot. Assuming that the 3D coordinates corresponding to the time t1 and t2 when the left foot touches the ground are L1(XL1, YL1, ZL1) and L2(XL2, YL2, ZL2), the step length calculation formula is:

[0104] Step length=||L2-L1||,

[0105]

[0106] The step frequency is the number of steps completed per unit time. Assuming that N steps are completed in time T, the step frequency calculation formula is: step frequency=N / T*60.

[0107] The model training unit is based on a loss function, assuming that the prediction result of the model is The true label is y, and the loss function L can use mean square error or cross-entropy loss.

[0108] Mean square error:

[0109] Cross-entropy loss:

[0110] The task of the diagnostic reasoning unit is to use the trained model to diagnose and analyze new gait data, and to predict the results.

[0111] Assuming that the input of the model is X, the weight is W and the bias is b, the calculation formula of the output layer is: z = WX + b, where z is the unactivated output.

[0112] Softmax activation is performed on the output to obtain the class probability distribution p:

[0113] Where p i is the probability of the ith class.

[0114] According to the probability distribution p, the class with the maximum probability is taken as the diagnostic result:

[0115] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. An intelligent gait analysis-assisted diagnostic system for knee joint function, characterized in that, It includes a data acquisition module, a 3D human posture modeling module, a gait feature extraction module, a gait feature measurement and analysis module, a deep learning analysis and diagnosis module, an interpretive model and user interaction module, a system integration module, and a data management and storage module; The data acquisition module collects human gait video data through a multi-view camera system; the 3D human posture modeling module converts key point data of the human body in 2D images into 3D coordinates to construct a 3D human posture model; the gait feature extraction module extracts multi-dimensional features related to gait from the multi-view video data; the gait feature measurement and analysis module performs precise quantitative measurement and analysis of the extracted gait features; the deep learning analysis and diagnosis module uses a deep learning model to analyze gait data to assist medical personnel in making accurate diagnoses; the interpretability model and user interaction module provides interpretability of the model reasoning process and evidence, providing medical personnel with tools to understand and verify diagnostic results; the system integration module integrates all modules to form a complete gait analysis and auxiliary diagnosis system for functional verification; and the data management and storage module manages and stores the collected gait data and analysis results. The data acquisition module includes a multi-view camera unit, a marker point reference unit, and a time alignment unit; the multi-view camera unit captures human gait videos from multiple angles, enabling it to capture complete gait information from different perspectives. During the initial data acquisition, the marker reference unit, under the guidance of medical experts, pastes markers on 2D key points of the human body to provide a reference for the manual data annotation of 2D key points; the time alignment unit performs time alignment on image frames captured by different cameras. The 3D human posture modeling module includes a 2D key point calibration unit, a parallax measurement unit, and a 3D modeling unit. The 2D key point calibration unit accurately calibrates the 2D key points of the human skeleton in the 2D image. The parallax measurement unit uses the parallax information of multi-view images to calculate and reconstruct 3D coordinates. The 3D modeling unit generates a 3D posture model of the human body based on kernel geometry theory and camera imaging principles. The task of the disparity measurement unit is to calculate the disparity of 2D key points based on multi-view images and then perform 3D reconstruction. The steps are as follows: Step 1: Input multi-view images: Obtain view images of the same scene. and ; Step 2: Feature Point Detection and Matching: In the image and Detect feature points in the middle; {P1}=DetectFeatures(I1),{P2}=DetectFeatures(I2) Step 3: Match feature point pairs; {(P1, P2)}=MatchFeatures({P1}, {P2}) Step 4: Calculate the fundamental matrix F: Calculate the fundamental matrix F using the matched feature point pairs. =0， Step 5: Calculate the epipolar lines: Calculate the epipolar lines of the feature points in the image from another viewpoint. ， ； Step 6: Calculate the disparity: Calculate the disparity based on the epipolar constraint, d=||P1-P2||.

2. The intelligent gait analysis auxiliary diagnostic system for knee joint function according to claim 1, characterized in that, The gait feature extraction module includes a joint angle extraction unit, a gait parameter extraction unit, and a spatiotemporal feature extraction unit. The gait parameter extraction unit extracts gait feature parameters such as stride length, stride frequency, and walking speed. The spatiotemporal feature extraction unit extracts the spatiotemporal features of gait based on spatiotemporal graph neural network technology, capturing the dynamic changes of gait over time. The joint angle extraction unit extracts knee joint angle change data from the 3D posture model, including flexion / extension angle, internal rotation / external rotation angle, and varus / valgus angle; The gait feature measurement and analysis module includes a gait parameter measurement unit, a feature analysis unit, and a medical analysis unit; the gait parameter measurement unit performs precise quantitative measurement of the extracted gait parameters; The feature analysis unit analyzes gait feature data and provides a gait feature change report; The medical analysis unit combines clinical knowledge to analyze the medical significance of gait data, providing support for diagnosis.

3. The intelligent gait analysis auxiliary diagnostic system for knee joint function according to claim 1, characterized in that, The deep learning analysis and diagnosis module includes a model training unit, a feature modeling unit, and a diagnostic inference unit. The model training unit trains a deep learning model using a large-scale gait dataset. The feature modeling unit models features from gait data to identify abnormal gait patterns. The diagnostic inference unit performs diagnostic analysis on new gait data based on the trained model. The interpretive model and user interaction module includes a reasoning and interpretation unit, a user interaction interface unit, and a feedback and adjustment unit; The reasoning and interpretation unit provides the reasoning process and basis for the diagnostic results, enhancing the interpretability of the model; The user interface unit is designed with a user-friendly interface to display gait analysis results and explanations, and supports user queries and interactions; the feedback and adjustment unit receives user feedback and adjusts the model and analysis strategy.

4. The intelligent gait analysis auxiliary diagnostic system for knee joint function according to claim 1, characterized in that, The system integration module includes an interface integration unit, a system interface unit, and a performance verification unit. The interface integration unit integrates various functional modules to ensure seamless connection between data and functions. The system interface unit constructs a complete system interface, supporting data acquisition, processing, analysis, and result display. The performance verification unit performs overall performance testing and verification of the system. The data management and storage module includes a data storage unit, a data retrieval unit, and a data security unit. The data storage unit securely stores gait videos, 3D posture models, gait feature data, and diagnostic results. The data retrieval unit provides efficient data retrieval and query functions, supporting quick user access. The data security unit ensures data security and privacy, preventing data leakage and unauthorized access.

5. The intelligent knee joint function gait analysis auxiliary diagnostic system according to claim 1, characterized in that, The task of the 2D keypoint calibration unit is to identify and calibrate the coordinates of 2D keypoints of the human body in the image. The algorithm includes a keypoint detection method based on a convolutional neural network, and its main formulas and steps are as follows: Input image I, extract features through convolutional neural network and generate keypoint heatmap H, with the heatmap corresponding to each keypoint as follows: ,in, This is the heatmap of the i-th key point; Key point coordinates are extracted from the peak positions of the heatmap. : The formula is expressed in the heat map. The mid-peak position, i.e., the coordinates of the key point.

6. The intelligent gait analysis auxiliary diagnostic system for knee joint function according to claim 2, characterized in that, The task of the gait parameter extraction unit is to extract gait-related parameters from 3D posture data. Stride length is the distance between adjacent feet touching the ground. Assuming the 3D coordinates corresponding to the times t1 and t2 when the left foot touches the ground are L1(XL1, YL1, ZL1) and L2(XL2, YL2, ZL2), the stride length calculation formula is: , , Step frequency is the number of steps completed per unit of time. Assuming N steps are completed in time T, the formula for calculating step frequency is: Step frequency = N / T * 60.

7. The intelligent knee joint function gait analysis auxiliary diagnostic system according to claim 3, characterized in that, The model training unit is based on a loss function, assuming the model's prediction result is... The real label is The loss function L can be either mean squared error or cross-entropy loss; Mean square error: , Cross-entropy loss: .

8. The intelligent knee joint function gait analysis auxiliary diagnostic system according to claim 3, characterized in that, The task of the diagnostic reasoning unit is to use the trained model to diagnose and analyze new gait data and predict the results. Assuming the model's input is X, the weights are W, and the bias is b, the formula for calculating the output layer is: z = WX + b, where z is the inactive output; Applying soft maximum activation to the output yields the class probability distribution p: ,in, It is the probability that it belongs to the i-th class; Based on the probability distribution p, the category with the highest probability is taken as the diagnostic result: .

Images