Diagnosis auxiliary system based on gait analysis and image recognition technology

Abstract

The invention belongs to the field of auxiliary diagnosis, and discloses a gait analysis and image recognition technology-based diagnosis auxiliary system, which comprises a wearable sensor module, a video shooting module, a recognition module and an auxiliary diagnosis module, the wearable module comprises a plurality of sensors used for acquiring monitoring data of the position where the wearable module is located and sending the monitoring data to the recognition module and the image shooting module. The video shooting module comprises a control unit, a shooting unit and a transmission unit; the control unit is used for acquiring a shooting frame rate based on the monitoring data; the shooting unit is used for shooting the subject based on the shooting frame rate to obtain a video containing action information of the subject; the transmission unit is used for sending the video obtained by the shooting unit to the identification module; the identification module is used for identifying the monitoring data video and acquiring a first action parameter and a second action parameter of the action of the subject; the auxiliary diagnosis module is used for obtaining an auxiliary diagnosis result based on the first action parameter and the second action parameter.

Description

Technical Field

[0001] The present invention relates to the field of auxiliary diagnosis, and in particular to a diagnosis auxiliary system based on gait analysis and image recognition technology. Background Art

[0002] It is a common diagnostic method to diagnose the rehabilitation of patients and other subjects through gait analysis. Common gait analysis methods include sensor-based analysis methods and image-based analysis methods. The sensor-based analysis method obtains the subject's joint position, movement range and other motion parameters based on the acceleration and other data of the sensors by having the subject wear multiple sensors. The image-based analysis method uses a shooting device to obtain a video containing the subject's movements when the subject makes movements such as walking, and uses an image recognition algorithm to obtain the subject's motion parameters. By comparing the obtained motion parameters with the standard motion parameters, the degree of rehabilitation of the subject can be known, and corresponding suggestions can be made to the subject's movements during rehabilitation training based on the differences between the motion parameters to improve the accuracy of the subject's movements during rehabilitation.

[0003] In order to improve the accuracy of the identified motion parameters, there are technologies in the prior art that use both sensor-based and image-based methods to obtain the subject's motion parameters. In this way, when the two methods identify the same motion parameters, it means that the current recognition result is more reliable. If the motion parameters identified by the two methods are different, the final identified motion parameters can be obtained by manual review by experts.

[0004] However, this recognition method using two recognition methods does not have effective linkage, and is prone to collecting too many images when the subject is still or making invalid movements, so that subsequent recognition requires a long time to complete the recognition of frames in the video to obtain action parameters. Summary of the invention

[0005] The purpose of the present invention is to disclose a diagnosis assistance system based on gait analysis and image recognition technology to solve the technical problems raised in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] The present invention provides a diagnosis assistance system based on gait analysis and image recognition technology, comprising a wearable sensor module, a video shooting module, a recognition module and an auxiliary diagnosis module;

[0008] The wearable module includes a plurality of sensors mounted on the subject's body, the sensors are used to obtain monitoring data of their own locations and send the monitoring data to the recognition module;

[0009] The sensor is also used to send monitoring data to the image capture module;

[0010] The video shooting module includes a control unit, a shooting unit and a transmission unit;

[0011] The control unit is used to obtain a shooting frame rate based on monitoring data;

[0012] The shooting unit is used to shoot the subject based on the shooting frame rate to obtain a video containing the subject's action information;

[0013] The transmission unit is used to send the video obtained by the shooting unit to the recognition module;

[0014] The recognition module is used to recognize the monitoring data and obtain the first action parameter of the subject's action, and is used to recognize the video sent by the transmission unit and obtain the second action parameter of the subject;

[0015] The auxiliary diagnosis module is used to obtain an auxiliary diagnosis result based on the first action parameter and the second action parameter.

[0016] Preferably, the plurality of sensors mounted on the subject's body include sensors mounted on the wrists, ankles, head, knees, and elbows.

[0017] Preferably, the monitoring data includes angular velocity and acceleration.

[0018] Preferably, sending the monitoring data to the image capture module includes:

[0019] The monitoring data is sent to the image capture module based on an adaptive sending interval.

[0020] Preferably, the adaptive sending interval is determined in the following manner:

[0021] For the tth sending interval, the calculation formula is:

[0022]

[0023] sedtim t and sedtim t-1 They represent the tth and t-1th sending intervals, respectively, and motamp t-1 and motamp t-2 Respectively represent the time interval itrl t-1 and time interval itrl t-2 The action trend value calculated from the monitoring data obtained in sedtimst Indicates the set sending interval. t-1 Less than motamp t-2 When case=1; when motamp t-1 Greater than or equal to motamp t-2 , case=0.

[0024] Preferably, the time interval itrl t-1 The starting time is tcal t-2 , the end time is tcal t-2 +sedtim t-2 , tcal t-2 Indicates the calculation of sedtim t-2 moment, sedtim t-2 Indicates the t-2th sending interval;

[0025] Time interval itrl t-2 The starting time is tcal t-1 , the end time is tcal t-1 +sedtim t-1 , tcal t-1 Indicates the calculation of sedtim t-1 moment, sedtim t-1 Indicates the t-1th sending interval.

[0026] Preferably, the calculation process of the action trend value includes:

[0027] Get each sensor in the time interval itrl z The maximum angular velocity obtained is angvel max and minimum angvel min ; z∈{t-1,t-2};

[0028] Calculate the action trend value:

[0029]

[0030] trendval z Indicates that based on the time interval itrl z The action trend value calculated from the monitoring data obtained in the calculation, angu represents the collection of all sensors, angvel max,i and angvel min,i They represent sensor i in the time interval itrl z The maximum and minimum angular velocity values ​​obtained in accui represent the sensor i in the time interval itrl zThe collection of accelerations obtained in accui, accj represents the acceleration in accui, acc ave Indicates the average value of acceleration in accui, accdev max Indicates that among all sensors, in the time interval itrl z The maximum value of the standard deviation of the acceleration obtained in , ns represents the total number of sensors, and λ represents the weight.

[0031] Preferably, the monitoring data is sent to the image capture module based on an adaptive sending interval, including:

[0032] After the t-1th sending interval ends, the acquired monitoring data is sent to the image capture module. After the sending is completed, the tth sending interval is immediately calculated. After the tth calculation interval is calculated, the tth sending interval is immediately entered.

[0033] Preferably, sending the acquired monitoring data to the image capture module includes:

[0034] The monitoring data obtained by each sensor for the most recent M times is sent to the image capture module.

[0035] Preferably, obtaining the shooting frame rate based on the monitoring data includes:

[0036] Use the following formula to calculate the shooting frame rate:

[0037]

[0038] frarat indicates the shooting frame rate, aveangvel i Represents the average angular velocity obtained by sensor i for the last M times, aveangvel max Indicates the maximum value of the average value of the angular velocity obtained by all sensors in the last M times, rat st Indicates the set frame rate; fradif indicates the dth frame fra d and the dDth frame fra D-d limb fluctuation area between; fraarea represents the area of ​​the frame; γ1 represents the first ratio, and γ2 represents the second ratio.

[0039] Beneficial effects:

[0040] In the process of performing gait analysis on the subject, the present invention links the data acquisition process of the sensor-based analysis method and the image-based analysis method, so that the video shooting frame rate can be adaptively changed with the change of the monitoring data obtained by the sensor. In this way, it is possible to avoid the frame rate of the video shot when the subject makes an invalid action being too high, thereby effectively reducing the total number of frames contained in the final video while maintaining the effective action in the video, thereby improving the efficiency of obtaining auxiliary diagnosis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of a diagnosis assistance system based on gait analysis and image recognition technology of the present invention.

[0043] Figure 2 Schematic diagram of the training process of the first deep learning model of the present invention. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0045] like Figure 1 In one embodiment shown, the present invention provides a diagnosis assistance system based on gait analysis and image recognition technology, including a wearable sensor module, a video shooting module, a recognition module and an auxiliary diagnosis module;

[0046] The wearable module includes a plurality of sensors mounted on the subject's body, the sensors are used to obtain monitoring data of their own locations and send the monitoring data to the recognition module;

[0047] The sensor is also used to send monitoring data to the image capture module;

[0048] The video shooting module includes a control unit, a shooting unit and a transmission unit;

[0049] The control unit is used to obtain a shooting frame rate based on monitoring data;

[0050] The shooting unit is used to shoot the subject based on the shooting frame rate to obtain a video containing the subject's action information;

[0051] The transmission unit is used to send the video obtained by the shooting unit to the recognition module;

[0052] The recognition module is used to recognize the monitoring data and obtain the first action parameter of the subject's action, and is used to recognize the video sent by the transmission unit and obtain the second action parameter of the subject;

[0053] The auxiliary diagnosis module is used to obtain an auxiliary diagnosis result based on the first action parameter and the second action parameter.

[0054] In the process of gait analysis of the subjects, the data acquisition processes of the sensor-based analysis method and the image-based analysis method are linked, so that the video shooting frame rate can be adaptively changed as the monitoring data obtained by the sensor changes. In this way, it is possible to avoid the frame rate of the video shot being too high when the subjects make invalid movements. While maintaining the effective movements in the video, the total number of frames contained in the final video is effectively reduced, thereby improving the efficiency of obtaining auxiliary diagnosis results.

[0055] In the present invention, the sensor is worn on the subject before the subject undergoes rehabilitation training, and monitoring data is acquired during the subject's rehabilitation training.

[0056] Preferably, identifying the monitoring data to obtain the first motion parameter of the subject's motion includes:

[0057] The monitoring data is input into a first deep learning model that has been trained in advance for recognition, and the first action parameters of the subject at each moment of monitoring data collection are obtained.

[0058] Specifically, Figure 2 As shown, the training process of the first deep learning model includes:

[0059] 1. Get the training set

[0060] ① Obtain by yourself: Sensor arrangement: Arrange sensors such as accelerometers, gyroscopes, and magnetometers in different parts of the human body (such as wrists, waist, ankles, etc.).

[0061] Data acquisition: Collect the three-axis acceleration, angular velocity and other related data of each sensor. The frequency of data acquisition is usually between 50Hz and 200Hz to capture sufficiently detailed motion information.

[0062] ② Obtain from public databases

[0063] 2. Data Preprocessing

[0064] Denoising: Use filters (such as low-pass filters or Kalman filters) to remove noise from sensor data.

[0065] Coordinate system transformation: If necessary, the sensor data can be transformed into a global coordinate system or a local coordinate system to unify the data reference system of different sensors.

[0066] Data segmentation: Split the continuous sensor data into windows of fixed length (e.g., one window per second), and the windows can overlap.

[0067] 3. Feature extraction

[0068] Time domain characteristics: Calculate the statistical characteristics of the original signal, such as mean, variance, kurtosis, skewness, etc.

[0069] Frequency domain features: Use Fourier transform to extract frequency domain features, such as main frequency, spectrum energy, etc.

[0070] Spatial features: If sensors are located at multiple body parts, the correlation and relative position changes between different parts can be calculated.

[0071] Advanced features: Calculate attitude angle and movement direction based on attitude models (such as quaternions and Euler angles).

[0072] 4. Model selection

[0073] LSTM (Long Short-Term Memory Network): Suitable for processing time series data and can capture long-term dependencies.

[0074] CNN (Convolutional Neural Network): It can extract local features on the time axis and has a good effect on pattern recognition.

[0075] Hybrid model: Combining the characteristics of LSTM and CNN, extracting features through the convolutional layer, and then performing sequence modeling through LSTM.

[0076] You can choose one of the above models for training.

[0077] 5. Model Training

[0078] Data labeling: Labeling the collected data, such as labeling posture types based on videos or other reference methods.

[0079] Divide the training set and test set: Ensure the diversity and representativeness of the data set. The commonly used division method is 70% for training and 30% for testing.

[0080] Choose loss function and optimizer: For classification tasks, the cross entropy loss function is often used; for regression tasks, the mean square error (MSE) is often used. The optimizer is usually Adam or SGD.

[0081] Training process: Iteratively train the model and adjust hyperparameters (such as learning rate, batch size) to improve the accuracy and robustness of the model.

[0082] Preferably, the method for identifying the video sent by the transmission unit and obtaining the second action parameter of the subject includes:

[0083] Split the video into multiple frames;

[0084] Each frame is input into the pre-trained second deep learning model for recognition to obtain the subject's action parameters in each frame.

[0085] Specifically, the training process of the second deep learning model includes:

[0086] 1. Data preprocessing

[0087] Data collection: First, a labeled dataset containing human gait is needed. Commonly used datasets include COCO, MPII, etc.

[0088] 2. Data enhancement: In order to improve the robustness of the model, the image can be enhanced by performing data enhancement, such as rotation, scaling, and cropping.

[0089] 3. Choose the right model architecture

[0090] Common pose estimation model architectures include:

[0091] OpenPose: This is one of the earliest multi-person pose estimation models, which locates joint points by gradually refining feature maps.

[0092] Hourglass Networks: Improving the detection capabilities of various parts of the human body through multi-scale feature learning.

[0093] Simple Baselines: A simple and effective pose estimation method based on convolutional neural network (CNN).

[0094] HRNet: A high-resolution network that can simultaneously retain high-resolution information and achieve higher accuracy.

[0095] You can choose one of the above models to train as the second deep learning model.

[0096] 4. Model training

[0097] Loss function: During the training process, the commonly used loss function is the mean square error (MSE), which is used to measure the difference between the predicted key point positions and the actual annotations.

[0098] Optimizer: Commonly used optimizers are Adam or SGD, which are used to update the model weights to minimize the loss.

[0099] Preferably, the action parameters include the positions of each skeletal node.

[0100] In one embodiment, the skeletal nodes include wrists, ankles, head, knees, and elbows.

[0101] In another embodiment, the skeletal nodes include wrists, ankles, cervical vertebrae, knees, and elbows.

[0102] Preferably, obtaining the auxiliary diagnosis result based on the first action parameter and the second action parameter includes:

[0103] The first step is to determine whether there is a corresponding frame at the acquisition time s for the first action parameter obtained at the acquisition time s. If so, the second action parameter corresponding to the frame is used as the second action parameter corresponding to the acquisition time s and the third step is entered. If not, the second step is entered.

[0104] The second step is to obtain the frame whose shooting time is closest to the acquisition time s, and use the second action parameter corresponding to the frame as the second action parameter corresponding to the acquisition time s, and then proceed to the third step;

[0105] The third step is to calculate the first similarity between the first action parameter obtained at the collection time s and the second action parameter corresponding to the collection time s;

[0106] The fourth step is to determine whether the first similarity is greater than the set first similarity threshold, if so, proceed to the fifth step, if not, exit the calculation;

[0107] Step 5, obtaining the gait type of the action corresponding to the first action parameter;

[0108] Step 6: Obtain standard motion parameters corresponding to the gait type;

[0109] Step 7, calculating the second similarity between the standard action parameters and the first action parameters;

[0110] In the eighth step, if the second similarity is greater than the set second similarity threshold, the auxiliary diagnosis result is that the subject's gait movement is standard; if the second similarity is less than the set second similarity threshold, the auxiliary diagnosis result is that the subject's gait movement is not standard.

[0111] In addition, after determining that the gait movement is not standard, the position of each bone point obtained based on the movement parameters can be compared with the position of the same bone point in the standard movement, and the distance between the two positions can be calculated, so that the distance can be provided to the subject to automatically provide relevant improvement suggestions for the subject's rehabilitation training process.

[0112] Preferably, the gait types include normal gait, tiptoe gait, striding gait, jogging gait, sprinting gait and the like.

[0113] Normal gait: This refers to the natural gait of healthy people without any external interference or internal disease. The stride is even and the rhythm is consistent. It is characterized by a stable gait, alternating movements of the two feet, moderate stride and constant speed.

[0114] Tiptoe gait: Patients walk mainly on their toes, which is common in patients with cerebral palsy.

[0115] Crossover gait: Because the toes cannot be lifted normally, the patient lifts the leg too high when walking, which is common in patients with common peroneal nerve injury.

[0116] Jogging gait: brisk steps, larger strides, and faster tempo.

[0117] Sprint gait: longer stride, faster pace, and more forefoot contact.

[0118] Preferably, the first similarity threshold is set to 0.8.

[0119] Preferably, the second similarity threshold is set to 0.9.

[0120] Preferably, the plurality of sensors mounted on the subject's body include sensors mounted on the wrists, ankles, head, knees, and elbows.

[0121] Specifically, the multiple sensors installed on the subject's body include sensors installed on the wrists of the left and right hands, sensors installed on the left ankle and right ankle, sensors installed on the forehead, sensors installed on the left knee and right knee, and sensors installed on the left elbow and right elbow.

[0122] Preferably, the monitoring data includes angular velocity and acceleration.

[0123] Preferably, sending the monitoring data to the image capture module includes:

[0124] The monitoring data is sent to the image capture module based on an adaptive sending interval.

[0125] If the acquired monitoring data is sent directly to the image capture module, on the one hand, the pressure of data transmission will be too great, because the sensor not only needs to transmit the data to the recognition module, but also needs to transmit the data to the video capture module. On the other hand, it will cause the shooting frame rate of the shooting unit to be switched too frequently, which affects the continuity between the frames in the video and is not conducive to obtaining accurate gait recognition results.

[0126] Preferably, the adaptive sending interval is determined in the following manner:

[0127] For the tth sending interval, the calculation formula is:

[0128]

[0129] sedtim t and sedtim t-1 They represent the tth and t-1th sending intervals, respectively, and motamp t-1 and motamp t-2 Respectively represent the time interval itrl t-1 and time interval itrl t-2 The action trend value calculated from the monitoring data obtained in sedtim st Indicates the set sending interval. t-1 Less than motamp t-2 When case=1; when motamp t-1 Greater than or equal to motamp t-2 , case=0.

[0130] The sending interval of the present invention is not fixed, but can change based on the change of the action trend value. In this way, when the action trend value becomes larger, the sending interval can be automatically reduced, so that the shooting unit can adjust the shooting frame rate more timely, so as to shoot more frames for effective actions, and ensure that the subsequent recognition process can have enough frames for analysis; and when the action trend value becomes smaller, the sending interval can be automatically expanded to reduce the probability of the occurrence of too many frames for invalid actions, and effectively improve the efficiency of the subsequent recognition process. Moreover, the change of the sending interval of the present invention is not a large fluctuation change, but a change based on the previous sending interval, so that the frame rate of the video obtained by the shooting unit changes more slowly, and when analyzing based on adjacent frames, it can ensure that the time interval of the adjacent frames obtained will not be too large, which is more valuable for reference. In addition, the amplitude of the change of the time interval is not fixed, but is closely related to the amplitude of the change between the two action trend values, thereby improving the correlation between the sending interval and the action trend value, and obtaining a sending interval with better adaptive effect.

[0131] Preferably, the sending interval is set to 5 seconds.

[0132] Preferably, the time interval itrl t-1 The starting time is tcal t-2 , the end time is tcal t-2 +sedtim t-2 , tcal t-2 Indicates the calculation of sedtim t-2 moment, sedtim t-2 Indicates the t-2th sending interval;

[0133] Time interval itrl t-2 The starting time is tcal t-1 , the end time is tcal t-1 +sedtim t-1 , tcal t-1 Indicates the calculation of sedtim t-1 moment, sedtim t-1 Indicates the t-1th sending interval.

[0134] Preferably, the value of t is greater than or equal to 3; the first sending interval and the second sending interval are both equal to the set sending interval.

[0135] Preferably, the calculation process of the action trend value includes:

[0136] Get each sensor in the time interval itrl z The maximum angular velocity obtained is angvel maxand minimum angvel min ; z∈{t-1,t-2};

[0137] Calculate the action trend value:

[0138]

[0139] trendval z Indicates that based on the time interval itrl z The action trend value calculated from the monitoring data obtained in the calculation, angu represents the collection of all sensors, angvel max,i and angvel min,i They represent sensor i in the time interval itrl z The maximum and minimum angular velocity values ​​obtained in accui represent the sensor i in the time interval itrl z The collection of accelerations obtained in accui, accj represents the acceleration in accui, acc ave Indicates the average value of acceleration in accui, accdev max Indicates that among all sensors, in the time interval itrl z The maximum value of the standard deviation of the acceleration obtained in , ns represents the total number of sensors, and λ represents the weight.

[0140] The action trend value of the present invention is calculated based on the two data of angular velocity and acceleration. When calculating, the present invention does not directly use the same method for the two data to obtain the result for weighting. Because in this way, the action trend value can only be calculated from a smaller number of types of data features, and the representativeness of the calculated results is worse than the representativeness of the results calculated by the present invention, which means that the adaptive effect of the sending interval obtained based on the calculated results is not good enough. Therefore, the present invention adopts different methods to calculate the data features of the two types of data respectively, so that the action trend value can be effectively represented from more different angles, which is conducive to improving the adaptive effect of the sending interval obtained based on the calculated results.

[0141] Preferably, the value of λ is 0.6.

[0142] Specifically, among all sensors, in the time interval itrl z The maximum value of the standard deviation of acceleration obtained in the time interval itrl is calculated for each sensor separately. z The standard deviation of the acceleration obtained in , and the maximum value selected from all the standard deviations obtained.

[0143] Preferably, the monitoring data is sent to the image capture module based on an adaptive sending interval, including:

[0144] After the t-1th sending interval ends, the acquired monitoring data is sent to the image capture module. After the sending is completed, the tth sending interval is immediately calculated. After the tth calculation interval is calculated, the tth sending interval is immediately entered.

[0145] That is, between two adjacent sending intervals, there exists a time interval for the sensor to send monitoring data to the image capturing module and an interval for calculating the next sending interval.

[0146] Preferably, sending the acquired monitoring data to the image capture module includes:

[0147] The monitoring data obtained by each sensor for the most recent M times is sent to the image capture module.

[0148] By sending only a portion of the latest data, it is possible to avoid sending too much data and ensure the validity of the sent data.

[0149] Specifically, the value of M may be 5.

[0150] Preferably, obtaining the shooting frame rate based on the monitoring data includes:

[0151] Use the following formula to calculate the shooting frame rate:

[0152]

[0153] frarat indicates the shooting frame rate, aveangvel i Represents the average angular velocity obtained by sensor i for the last M times, aveangvel max Indicates the maximum value of the average value of the angular velocity obtained by all sensors in the last M times, rat st Indicates the set frame rate; fradif indicates the dth frame fra d and the dDth frame fra D-d The limb fluctuation area between d It indicates the frame closest to the imaging time and the time of receiving the monitoring data sent by the sensor; fraarea indicates the area of ​​the frame; γ1 indicates the first ratio, and γ2 indicates the second ratio.

[0154] In the present invention, the shooting frame rate is not only obtained based on the monitoring data, but also takes into account the change range of the position of the subject's limbs between frames in the video. Therefore, the shooting frame rate can be calculated from the data obtained in two different acquisition methods. When the average value of the angular velocity is larger and the limb fluctuation area is larger, the frame rate of the present invention is also larger. When the average value of the angular velocity is smaller and the limb fluctuation area is smaller, the frame rate of the present invention is also smaller. The shooting frame rate can be adaptively changed with the change of the angular velocity and the change of the limb fluctuation area, so that the calculated shooting frame rate is more effective, and the number of frames of invalid actions contained in the video can be more effectively reduced.

[0155] Preferably, the set frame rate may be 30 FPS.

[0156] Preferably, the value of D can be 10.

[0157] Preferably, the first ratio and the second ratio are 0.7 and 0.3 respectively.

[0158] Preferably, fra d and fra D-d The calculation process of the limb fluctuation area includes:

[0159] For fra d To identify, from fra d Get the body region p belonging to the subject d ;

[0160] For fra D-d To identify, from fra D-d Get the body region p belonging to the subject D-d ;

[0161] Get p d The set of pixel coordinates LU d ;

[0162] Get p D-d The set of pixel coordinates LU D-d ;

[0163] Get LU d In, not LU D-d The set of pixel points Px;

[0164] The pixel point in Px is in fra d The area occupied by the limb is taken as the limb fluctuation area.

[0165] The fluctuation area of ​​the limbs is calculated by the change amplitude of the subject's body area in frames with similar time. The effectiveness of the monitoring data obtained by the sensor can be judged. When the effectiveness is higher, that is, the correlation between the change process of the monitoring data and the change process of the limb fluctuation area is higher, the change degree of the shooting frame rate is greater, and the calculated change degree is more accurate.

[0166] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A diagnostic assistance system based on gait analysis and image recognition technology, It is characterized in that It includes a wearable sensor module, a video shooting module, an identification module and an auxiliary diagnosis module; The wearable module includes a plurality of sensors mounted on the subject's body, the sensors are used to obtain monitoring data of their own locations and send the monitoring data to the recognition module; The sensor is also used to send monitoring data to the image capture module; The video shooting module includes a control unit, a shooting unit and a transmission unit; The control unit is used to obtain a shooting frame rate based on monitoring data; The shooting unit is used to shoot the subject based on the shooting frame rate to obtain a video containing the subject's action information; The transmission unit is used to send the video obtained by the shooting unit to the recognition module; The recognition module is used to recognize the monitoring data and obtain the first action parameter of the subject's action, and is used to recognize the video sent by the transmission unit and obtain the second action parameter of the subject; The auxiliary diagnosis module is used to obtain an auxiliary diagnosis result based on the first action parameter and the second action parameter.

2. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 1, It is characterized in that Multiple sensors mounted on the subject's body include sensors mounted on the wrists, ankles, head, knees, and elbows.

3. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 1, It is characterized in that The monitoring data include angular velocity and acceleration.

4. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 3, It is characterized in that Send monitoring data to the image capture module, including: The monitoring data is sent to the image capture module based on an adaptive sending interval.

5. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 4, It is characterized in that The adaptive sending interval is determined as follows: For the tth sending interval, the calculation formula is: sedtim t and sedtim t-1 They represent the tth and t-1th sending intervals, respectively, and motamp t-1 and motamp t-2 Respectively represent the time interval itrl t-1 and time interval itrl t-2 The action trend value calculated from the monitoring data obtained in sedtim st Indicates the set sending interval. t-1 Less than motamp t-2 When case=1; when motamp t-1 Greater than or equal to motamp t-2 , case=0.

6. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 5, It is characterized in that Time interval itrl t-1 The starting time is tcal t-2 , the end time is tcal t-2 +sedtim t-2 , tcal t-2 Indicates the calculation of sedtim t-2 moment, sedtim t-2 Indicates the t-2th sending interval; Time interval itrl t-2 The starting time is tcal t-1 , the end time is tcal t-1 +sedtim t-1 , tcal t-1 Indicates the calculation of sedtim t-1 moment, sedtim t-1 Indicates the t-1th sending interval.

7. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 5, It is characterized in that The calculation process of the action trend value includes: Get each sensor in the time interval itrl z The maximum angular velocity obtained is angvel max and minimum angvel min ; z∈{t-1,t-2}; Calculate the action trend value: trendval z Indicates that based on the time interval itrl z The action trend value calculated from the monitoring data obtained in the calculation, angu represents the collection of all sensors, angvel max,i and angvel min,i They represent sensor i in the time interval itrl z The maximum and minimum angular velocity values ​​obtained in , accui represents the sensor i in the time interval itrl z The collection of accelerations obtained in accui, accj represents the acceleration in accui, acc ave Indicates the average value of acceleration in accui, accdev max Indicates that among all sensors, in the time interval itrl z The maximum value of the standard deviation of the acceleration obtained in , ns represents the total number of sensors, and λ represents the weight.

8. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 4, It is characterized in that The monitoring data is sent to the image capture module based on an adaptive sending interval, including: After the t-1th sending interval ends, the acquired monitoring data is sent to the image capture module. After the sending is completed, the tth sending interval is immediately calculated. After the tth calculation interval is calculated, the tth sending interval is immediately entered.

9. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 8, It is characterized in that The acquired monitoring data is sent to the image capture module, including: The monitoring data obtained by each sensor for the most recent M times is sent to the image capture module.

10. A diagnostic assistance system based on gait analysis and image recognition technology according to claim 8, It is characterized in that Get the shooting frame rate based on monitoring data, including: Use the following formula to calculate the shooting frame rate: frarat indicates the shooting frame rate, aveangvel i Represents the average angular velocity obtained by sensor i for the last M times, aveangvel max Indicates the maximum value of the average value of the angular velocity obtained by all sensors in the last M times, rat st Indicates the set frame rate; fradif indicates the dth frame fra d and the dDth frame fra D-d The limb fluctuation area between the two; fraarea represents the area of ​​the frame; γ 1 represents the first ratio, γ 2 Indicates the second ratio.

Images