Construction method of rehabilitation platform

Abstract

The invention discloses a construction method of a rehabilitation platform, which is characterized in that a treadmill capable of providing safety support is used for walking training, various data records are carried out through an image and motion capture device and a wearable monitoring device, and then deep learning analysis is carried out through a full-scene AI calculation framework. Motion capture data are recorded through image and motion capture equipment, development engines such as U3D / UE4 are connected in real time, a real-time motion state is presented on a display screen by applying a virtual human real-time driving and rendering technology, and virtual human real-time driving height matching is achieved. A virtual online platform is constructed, and a subject enters the platform as a virtual human identity to perform three-dimensional interaction including online treatment, online treatment and the like. Meanwhile, data of various data records can be recorded through image and motion capture equipment, after deep learning is carried out through a full-scene AI calculation framework, data butt joint with an illness state database of a subject is carried out, deep learning is further carried out, a knowledge graph is constructed, and disease prediction, body state monitoring and the like can be carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of network platforms, and in particular to a method for constructing a rehabilitation platform. Background Art

[0002] Since 2018, with the increasing demand for deep learning applications in the industry, the development of deep learning frameworks has entered a new era, marked by the emergence of a new generation of frameworks represented by Google's open source Jax, TensorFlow2.x, and Huawei's open source Mindspore.

[0003] The model algorithm implementation of computer vision and natural language processing has become increasingly mature, and the focus of framework development has also shifted accordingly. On the basis of previous multiple fields, further attention is paid to full-scenario capabilities (that is, full-dimensional coverage of various training development, reasoning deployment, industrial application and other scenarios) and more efficient computing capabilities. Industry progress such as super-large models represented by GPT-3 and field innovations represented by AIphaFold2 reflect the main trend of in-depth integration of Al and scientific computing. Mathematical modeling and quantitative solutions are the main tasks of scientific computing. The rapid development of artificial intelligence technology has profoundly affected fields such as machine vision and natural language processing, while also providing a new paradigm for scientific computing.

[0004] AI methods can directly perform computable modeling of the problem to be solved based on data generated by experiments or calculations, or they can directly couple control equations and boundary conditions to obtain effective solutions to complex problems without supervised data. Combining data with physical laws also demonstrates extraordinary capabilities in areas such as forward solving, inverse problems, and data assimilation. Simulation toolkits represented by SimNet and SciML deeply integrate cutting-edge technologies in scientific computing and AI, and have gradually been applied in many fields of scientific inquiry and engineering applications, but there are still very few applications in the field of medical clinical practice. AI development in the field of rehabilitation is still a blue ocean. Summary of the invention

[0005] The present invention provides a method for constructing a rehabilitation platform, aiming to solve the above-mentioned problems existing in the prior art.

[0006] The technical solution provided by the present invention is as follows:

[0007] A method for constructing a rehabilitation platform comprises the following steps:

[0008] Provide a device or place for safety support, conduct limb movement, coordination, balance or walking training for the subject, collect data from the subject through images, motion capture equipment, wearable detection equipment and environmental sensor equipment to obtain first data, pre-process and feature extract the first data in sequence to obtain second data, connect the first data with the disease database, and determine the mapping relationship between the first data and the historical data in the disease database;

[0009] A deep learning model is constructed, the second data is used as input to the deep learning model, and third data is output, wherein the third data includes the evaluation results of the subject's limb function status, the prediction of rehabilitation training effects, and the identification of abnormal movement patterns.

[0010] Furthermore, the constructing of the deep learning model, taking the second data as input to the deep learning model, and outputting the third data, further includes:

[0011] The deep learning model is trained by annotating data, the parameters of the model are continuously adjusted, the loss function of the model is optimized, and the prediction accuracy of the model is improved. The annotated data includes the evaluation results of the limb function status of the subject by a professional rehabilitation therapist, the rehabilitation training effect, and the actual observation data;

[0012] Identify and extract key entities from the data processed by the deep learning model, determine the relationships between the extracted key entities, build a knowledge graph system, and update and optimize the knowledge graph system;

[0013] The first data is integrated with the historical data in the disease database and input into the trained deep learning model and the knowledge graph system to perform disease prediction and physical condition monitoring on the subjects.

[0014] The step of constructing the deep learning model, taking the second data as input to the deep learning model, and outputting the third data further includes:

[0015] Furthermore, the third data is connected to the U3D / UE4 development engine in real time, and the real-time driving and rendering technology of the virtual human is used to present the real-time motion state on the display screen, so as to achieve a high degree of matching of the real-time driving of the virtual human;

[0016] A virtual online platform is constructed, and subjects enter the platform as virtual people to conduct three-dimensional interaction, which includes online medical consultation and online treatment.

[0017] Furthermore, the preprocessing and feature extraction of the first data in sequence comprises the following steps:

[0018] Cleaning the image data in the first data, removing interference factors, and performing image standardization processing to ensure the consistency of image data from different sources;

[0019] Converting and calibrating the motion capture data obtained by the motion capture device in the first data into a format suitable for processing by a deep learning framework, and correcting the motion capture data according to the calibration parameters of the device to improve the accuracy of the data;

[0020] Filtering the physiological data recorded by the wearable monitoring device in the first data to remove abnormal values ​​and baseline drift to make the data smoother and more stable, wherein the physiological data includes heart rate, blood pressure, and muscle electrical signals;

[0021] Using a convolutional neural network to extract features from the image data;

[0022] Extracting joint angle, joint velocity, joint acceleration and motion time features from the motion capture data;

[0023] The time domain characteristics of heart rate variability, exercise intensity and muscle electrical signal characteristics are extracted from the physiological data.

[0024] Furthermore, the historical data includes the patient's case information, diagnosis results, and treatment process records;

[0025] Determine the mapping relationship between the first data and the historical data in the disease database, specifically: use the subject's medical record number or unique identifier as an association key, and match the first data with the corresponding medical record information.

[0026] Furthermore, the construction of the deep learning model is specifically as follows:

[0027] The second data is processed using a convolutional neural network, a recurrent neural network or a long short-term memory network, and then the features of the second data in the fully connected layer are fused with the structured data in the disease database.

[0028] Furthermore, the deep learning model is trained by annotating data, the parameters of the model are continuously adjusted, the loss function of the model is optimized, and the prediction accuracy of the model is improved, specifically:

[0029] The deep learning model constructed is trained using the labeled data, and the parameters of the model are continuously adjusted through the back propagation algorithm to optimize the loss function of the model and improve the prediction accuracy of the model. During the training process, cross-validation is used to prevent overfitting and ensure that the deep learning model has good generalization ability for new data.

[0030] Furthermore, identifying and extracting key entities from the data processed by the deep learning model, determining the relationships between the extracted key entities, constructing a knowledge graph system, and updating and optimizing the knowledge graph system include the following steps:

[0031] Identify and extract key entities from the data processed by the deep learning model, the key entities including disease name, body part, symptom manifestation, treatment method, movement characteristics, and extract historical data of the subject from the disease database;

[0032] Extracting the relationship between the key entities and the historical data to construct a knowledge graph system;

[0033] The relationship between the newly discovered key entities and the historical data is added to the knowledge graph system, and the existing relationships are corrected and improved.

[0034] Furthermore, the joint velocity feature is extracted from the motion capture data using a first formula, which is:

[0035]

[0036] Where Δt is the time interval, θ(t) is the joint angle, t is the time, and v(t) is the joint velocity;

[0037] The joint acceleration is calculated for the motion capture data using a second formula, where the second formula is:

[0038]

[0039] Where a(t) is the joint acceleration;

[0040] The time domain feature extraction of heart rate variability is performed on the physiological data by using the third formula, and the third formula is:

[0041]

[0042] Among them, N is the heartbeat interval within a period of time, RR n is the heartbeat interval sequence, n is a positive integer less than or equal to N, is the mean value of the heartbeat interval sequence, and SDNN is the standard deviation.

[0043] Furthermore, muscle electrical signal features are extracted from the physiological data using a fourth formula, and the fourth formula is:

[0044]

[0045] Wherein, x(n) is a muscle electrical signal sequence, n=0, 1, ..., N-1, and x(n) is transformed by FFT to obtain X(k).

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention provides a method for constructing a rehabilitation platform, which can provide a treadmill with safe support for walking training, record various data through images, motion capture devices and wearable monitoring devices, and then perform deep learning analysis through a full-scene AI computing framework. The motion capture data is recorded through images and motion capture devices, and the development engines such as U3D / UE4 are connected in real time. The real-time driving and rendering technology of virtual humans is used to present the real-time motion state on the display screen, and the real-time driving of virtual humans is highly matched. A virtual online platform is constructed, and the subjects enter the platform as virtual humans for three-dimensional interaction, including online medical consultation and online treatment. At the same time, the data recorded by various data through images and motion capture devices can be deeply learned through the full-scene AI computing framework, and then the data is connected with the subject's condition database, and further deep learning is performed to construct a knowledge graph, which can be used for disease prediction and physical condition monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a framework diagram of the construction method of the rehabilitation platform of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described below are part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0050] See also Figure 1 The present invention provides a method for constructing a rehabilitation platform, comprising the following steps:

[0051] S1. Provide a device or place for safety support, perform limb movement, coordination, balance or walking training on the subject, collect data on the subject through images, motion capture equipment, wearable detection equipment and environmental sensor equipment to obtain first data, pre-process and feature extract the first data in sequence to obtain second data, connect the first data with the disease database, and determine the mapping relationship between the first data and the historical data in the disease database.

[0052] Specifically, after the subjects have performed walking training and collected various data through images, motion capture devices and wearable monitoring devices, the data is first preprocessed. For image data, image cleaning is performed to remove interference factors such as noise and blur, and image standardization is performed, such as adjusting the image size, cropping to key areas, etc., to ensure the consistency of image data from different sources. The motion capture data obtained by the motion capture device is converted and calibrated to a format suitable for processing by the deep learning framework, and the data is corrected according to the calibration parameters of the device to improve the accuracy of the data. The data recorded by the wearable monitoring device (such as heart rate, blood pressure, muscle electrical signals and other physiological data) is filtered to remove outliers and baseline drift, making the data smoother and more stable.

[0053] The specific operations of data collection are as follows:

[0054] Image capture equipment: usually multiple cameras are used to shoot subjects from different angles, and the position information of the subjects in three-dimensional space is obtained by analyzing multiple perspective images based on the principle of stereoscopic vision. These devices can capture key points of the human body, such as joint positions (such as shoulders, elbows, wrists, hips, knees, ankles, etc.), and convert them into three-dimensional coordinate data. For example, when using an optical motion capture system, markers are attached to key parts of the human body. These markers reflect light of a specific wavelength, and the camera can identify and track the movement trajectory of these markers, and then calculate the spatial position and posture information of each part of the human body.

[0055] Motion capture equipment: In addition to optical motion capture, there are also inertial motion capture equipment, which uses sensors such as accelerometers, gyroscopes and magnetometers, which are directly installed on various parts of the human body to obtain real-time human motion information by measuring the acceleration, angular velocity and magnetic field information of the movement. The data from these sensors are integrated and the movement posture and position changes of the limbs are calculated based on the principles of kinematics.

[0056] S2. Build a deep learning model, use the second data as input to the deep learning model, and output third data, where the third data includes the evaluation results of the subject's limb function status, the prediction of rehabilitation training effects, and the identification of abnormal movement patterns.

[0057] Build a multimodal fusion deep learning model, and use the features extracted from images, motion capture, and wearable monitoring data as the input of the model. For example, a deep neural network (DNN) architecture can be used to input features of different modalities at different layers of the network, and feature fusion is performed through a fully connected layer.

[0058] The output of the model can include multiple aspects, such as the evaluation results of the subject's limb function status, the prediction of rehabilitation training effects, the identification of abnormal movement patterns, etc. For example, a limb function score is output to quantify the comprehensive level of the subject's current limb movement ability, balance ability, coordination, etc.; or to predict the trend of improving the effect of continuing the current rehabilitation training plan in the future.

[0059] Optionally, the method for constructing the rehabilitation platform mentioned in the present invention also includes the following method:

[0060] S3. Train the deep learning model by annotating data, continuously adjust the model parameters, optimize the model's loss function, and improve the model's prediction accuracy. The annotated data include the professional rehabilitation therapist's assessment results of the subject's limb function status, the rehabilitation training effect, and actual observation data.

[0061] S4. Identify and extract key entities from the data processed by the deep learning model, determine the relationship between the extracted key entities, build a knowledge graph system, and update and optimize the knowledge graph system.

[0062] S5. Integrate the first data with the historical data in the disease database and input them into the trained deep learning model and knowledge graph system to perform disease prediction and physical condition monitoring on the subjects.

[0063] Alternatively, the method for constructing the rehabilitation platform mentioned in the present invention also includes another method:

[0064] S6. Connect the third-party data to development engines such as U3D / UE4 in real time, use virtual human real-time driving and rendering technology to present real-time motion status on the display screen, and achieve a high degree of matching between virtual human real-time driving and virtual human.

[0065] S7. Construct a virtual online platform, and the subjects enter the platform as virtual people to conduct three-dimensional interaction. The three-dimensional interaction includes online medical consultation and online treatment.

[0066] In step S6, real-time connection with development engines such as U3D / UE4 includes:

[0067] Data transmission and interfaces:

[0068] Data transmission protocol: TCP / IP or UDP protocol is usually used to transmit the collected data from the motion capture device to the development engine. For example, the 3D coordinate data and posture data are encapsulated in a custom data message and sent to the server or local program where the development engine is located according to the protocol specification.

[0069] Development engine interface: Both U3D and UE4 provide APIs (application programming interfaces) to allow external data to be input. Taking Unity as an example, using C# scripts can receive data through Unity's network API, and for UE4, C++ or blueprint scripts can be used to receive data through network components. For example, using Unity, you can use the NetworkTransport class to receive network data.

[0070] Data conversion:

[0071] The data received from the motion capture device may need to be converted to fit the internal data structure of the development engine. For example, the original coordinates output by the motion capture device may be world coordinates, which need to be converted to the local coordinate system of the development engine. Let the original coordinates be (x w ,y w , z w ), the local coordinates of the development engine are (x l ,y l , z l ), if the offset of the origin of the local coordinate system of the development engine relative to the origin of the world coordinate system is (x0, y0, z0), the conversion formula is:

[0072] x l =x w -x0,

[0073] y l =y w -y0,

[0074] z l =z w -z0.

[0075] In step S6, the virtual human real-time driving and rendering technology includes a skeletal animation system:

[0076] Both U3D and UE4 have powerful skeletal animation systems. Taking UE4 as an example, a skeletal mesh is used to represent a virtual person, which consists of bones and meshes attached to the bones. After receiving the motion capture data, the data is mapped to the corresponding joints of the bones. Assuming that the rotation angle θ of a joint (such as the elbow joint) is obtained from the motion capture device, in UE4, it can be applied to the corresponding bones using code.

[0077] In step S6, the real-time drive is implemented by the following operations:

[0078] Motion smoothing: To avoid motion jitter, the received data can be filtered, for example, using a low-pass filter. Assuming the received joint position sequence is, the differential equation of the low-pass filter is y[n]=αy[n]+(1-α)x[n], where α is the filter coefficient, 0<α<1. This can make the motion smoother and more natural.

[0079] Posture matching and calibration: In order to make the virtual human's movements highly match the actual human movements, the motion capture device and the initial posture of the virtual human need to be calibrated. For example, when starting the system, let the subject pose in a standard T-pose, record the initial posture of the motion capture device and the virtual human, calculate the deviation matrix M between the two, and the subsequent received motion data need to be multiplied by this deviation matrix for calibration.

[0080] In step S6, the real-time motion status is displayed

[0081] The rendering principle is:

[0082] The development engine will render based on the virtual human's skeleton posture and mesh information, using lighting, texture and other information. During the rendering process, the lighting effect is calculated using a lighting model (such as the Phong lighting model), and the formula is:

[0083] I=I a k a +I d k d (L*N)+I s k s (R*V) n ,

[0084] Among them I a is the ambient light intensity, k a is the ambient light reflection coefficient, I d is the diffuse light intensity, k d is the diffuse reflectance, L is the light direction, N is the surface normal, I s is the intensity of specular light, k s is the specular reflection coefficient, R is the reflected light direction, V is the viewing direction, and n is the specular index.

[0085] In step S6, the high matching is achieved

[0086] Minimize Error:

[0087] The error between the actual motion and the virtual human motion is calculated by the least square method or other optimization algorithms to achieve a high degree of matching. For example, for the joint position, assuming that the actual joint position is (x r ,y r , z r ), the virtual human joint position is (xv ,y v , z v ), the error function can be defined as:

[0088]

[0089] The motion parameters of the virtual human are adjusted through the optimization algorithm to minimize E and achieve a high degree of matching.

[0090] The present invention proposes a method for constructing a virtual rehabilitation platform based on a full-scene AI computing framework, which can provide a treadmill with safe support for walking training, record various data through images, motion capture devices and wearable monitoring devices, and then perform deep learning analysis through the full-scene AI computing framework. At the same time, the motion capture data is recorded through images and motion capture devices, and the development engines such as U3D / UE4 are connected in real time. The real-time driving and rendering technology of virtual people is used to present the real-time motion state on the display screen, and the real-time driving of virtual people is highly matched. After that, a virtual online platform is constructed, and the subjects enter the platform as virtual people for three-dimensional interaction, including online medical consultation and online treatment. At the same time, the data recorded by images and motion capture devices can be deeply learned through the full-scene AI computing framework, and then the data is connected with the subject's disease database, and further deep learning is performed to construct a knowledge graph, which can be used for disease prediction and physical condition monitoring.

[0091] The full-scenario AI computing framework adopted by the present invention may include a new generation of frameworks represented by Google's open source Jax, TensorFlow2.x, and Huawei's open source Mindspore.

[0092] In step S1, the device used in the present invention includes a treadmill that can provide safety support, an image capture device, and a wearable monitoring device. The treadmill that can provide safety support includes a conveyor belt, a display screen, a weight loss safety device, and a remote data receiving device. The image and motion capture device can include the following categories: optical, inertial, mechanical, acoustic, and electromagnetic.

[0093] Optical motion capture is the process of capturing and locating objects on the scene through optical principles. Motion gesture capture is achieved by capturing the position of markers fixed on the human body or object through an optical lens. Optical motion capture relies on a complex set of optical cameras that use computer vision principles, and multiple high-speed cameras track the target feature points of the target from different angles to capture full-body movements. Optical motion capture can be divided into passive capture and active capture. This classification is different from markers. Active means that the marker is actively emitting light and can even be ID-coded so that the lens can observe the marker in its field of view by emitting its own light and record its movement trajectory to capture the marker. Passive optical motion capture emits infrared light of a specific wavelength through the light board of the lens itself, and illuminates the marker through the light board. The marker is specially treated with a reflection of the infrared light emitted by the lens light board, so that the lens can capture and record the movement trajectory in the field of view of the marker.

[0094] Inertial motion capture uses inertial navigation sensors AHRS and IMU to measure the acceleration, orientation and tilt angle of the captured person or object. Inertial motion capture requires a variety of wireless controls, battery packs, sensors and other accessories. Similar to a full-body suit, sensors in various parts of the human body capture data about the human body or object.

[0095] Mechanical motion capture systems rely on mechanical devices to track and measure motion trajectories. A typical system consists of multiple joints and rigid links, with angle sensors installed in rotatable joints to measure the rotation angles of the joints. When the device moves, the position and motion trajectory of the end point of the rod in space can be obtained based on the angle change measured by the angle sensor and the length of the link.

[0096] The acoustic motion capture system is generally composed of a transmitting device, a receiving system and a processing system. The transmitting device usually refers to an ultrasonic generator, and the receiving system usually consists of more than three ultrasonic probes. By measuring the phase difference or phase difference of the sound wave from the transmitting device to the sensor and determining the distance to the receiving sensor, the position and direction of the ultrasonic generator to the receiver is determined by the distance information obtained by the three triangular receiving sensors.

[0097] Electromagnetic motion capture systems generally consist of a transmitter, a receiving sensor, and a data processing unit. The transmitter generates an electromagnetic field distributed in space according to a certain time and space law; the receiving sensor is located at a key position in the performer's body, and when the performer passes through the electromagnetic field, the receiving sensor transmits the received signal to the processing unit via a cable or wirelessly, and the spatial position and direction of each sensor can be solved from the receiving sensor.

[0098] Optionally, in step S1, preprocessing and feature extraction are performed on the first data in sequence, including the following steps:

[0099] S101, cleaning the image data in the first data, removing interference factors, and performing image standardization processing to ensure consistency of image data from different sources.

[0100] The first data collected by images, motion capture devices, and wearable monitoring devices have different formats and data types. First, these first data need to be converted so that they can be uniformly processed by the full-scenario AI computing framework. For example, image data may be in JPEG or PNG format and need to be converted into an array format suitable for deep learning framework processing; the inertial data of the motion capture device may be in a specific sensor data format and needs to be converted into a general time series data format, etc.

[0101] Since the frequency and timestamps of data collected by different devices may be inconsistent, time synchronization is required. By recording accurate time information during data collection and using timestamps to align data from image, motion capture, and wearable monitoring devices, we can ensure that multimodal data at the same time point can accurately correspond to each other for subsequent comprehensive analysis.

[0102] The integrated raw data (i.e., the first data) is cleaned to remove outliers, noise data, and erroneous data caused by equipment failure or signal interference. For example, in optical motion capture, if a marker point is temporarily lost or misidentified, it needs to be processed through data interpolation or a correction algorithm based on adjacent frame data. For image data, image enhancement processing may be required, such as adjusting brightness and contrast, to improve the accuracy of feature extraction.

[0103] S102: converting and calibrating the motion capture data in the first data obtained by the motion capture device, converting the motion capture data into a format suitable for processing by a deep learning framework, and correcting the motion capture data according to the calibration parameters of the device to improve the accuracy of the data.

[0104] S103, filtering the physiological data recorded by the wearable monitoring device in the first data to remove abnormal values ​​and baseline drift, so as to make the data smoother and more stable, the physiological data including heart rate, blood pressure, and muscle electrical signals.

[0105] S104. Use a convolutional neural network to extract features from the image data.

[0106] Convolutional neural networks (CNNs) are used to extract features from image data. The convolutional layer of CNN can automatically learn local features in images, such as edges, textures, shapes, etc. For example, through convolution kernels of different sizes, multi-level features can be captured, ranging from subtle changes in muscle texture to the overall body posture contour.

[0107] Pre-trained CNN models (such as ResNet, VGG, etc.) are used and fine-tuned according to the characteristics of rehabilitation training data. These pre-trained models have learned a wealth of common image features on large-scale image datasets. Fine-tuning can quickly adapt them to image feature extraction tasks in rehabilitation scenarios, such as identifying specific rehabilitation postures, movement states of body parts, etc.

[0108] S105, extracting joint angles, joint velocities, joint accelerations and motion time features from the motion capture data.

[0109] Specifically, kinematic features such as joint angles, joint velocities, and joint accelerations are extracted from the motion capture data. These features can reflect the movement patterns and dynamic changes of the subjects' limbs during walking training. For example, the joint velocity is obtained by calculating the difference between the joint angles of adjacent time steps, and the change in velocity is further calculated to obtain the joint acceleration. These features are helpful in analyzing the smoothness, flexibility, and coordination of limb movements.

[0110] Extract time series features from action sequence data, such as using recurrent neural networks (RNN) or long short-term memory networks (LSTM). They can learn the dependencies of actions in the time dimension, capture the sequence and rhythm changes of actions, and thus identify different walking gait patterns (such as normal gait, hemiplegic gait, Parkinson's gait, etc.).

[0111] S106. Extracting time domain features of heart rate variability, exercise intensity, and muscle electrical signal features from physiological data.

[0112] Specifically, we extract sports-related features from physiological data, such as the correlation between heart rate variability (HRV) and exercise intensity. In rehabilitation training, changes in exercise intensity will cause changes in heart rate variability. By analyzing this correlation, we can evaluate the subject's physiological adaptability to exercise and fatigue level.

[0113] Extract the characteristics of muscle electrical signals, such as signal amplitude, frequency, time domain and frequency domain characteristics, etc. These characteristics can reflect the degree of muscle activation and fatigue state, which is of great significance for evaluating the impact of rehabilitation training on muscle function.

[0114] Optionally, the historical data mentioned in step S1 includes the patient's case information, diagnosis results, and treatment process records.

[0115] In step S1, a mapping relationship between the first data and the historical data in the disease database is determined, specifically: the medical record number or unique identifier of the subject is used as an association key to match the first data with the corresponding medical record information.

[0116] Before determining the mapping relationship between the first data and the historical data in the disease database, database connection and permission management are performed first: a secure connection channel with the subject's disease database is established to ensure the confidentiality and integrity of data transmission. A strict permission management mechanism is set up so that only authorized personnel and programs can access and operate the data. For example, an encrypted network connection protocol (such as SSL / TLS) is used, and access is restricted through multi-factor authentication methods such as username and password, digital certificates, or biometric recognition.

[0117] Then perform data mapping and association: determine the mapping relationship between the image, motion capture data and the data in the disease database. The disease database may contain the patient's case information (such as age, gender, medical history, etc.), diagnosis results, treatment process records, etc. Associate these data with the real-time motion data collected. For example, use the patient's medical record number or unique identifier as the association key to match the current motion data with the corresponding medical record information. In this way, when performing deep learning analysis, the patient's historical condition and current body movement status can be comprehensively considered.

[0118] Optionally, in step S2, constructing a deep learning model specifically includes: using a convolutional neural network, a recurrent neural network or a long short-term memory network to process the second data, and then fusing the features of the second data in the fully connected layer with the structured data in the disease database.

[0119] Specifically, the constructed deep learning model can integrate image, motion capture and disease data. For example, a convolutional neural network (CNN) can be used to process image data, a recurrent neural network (RNN) or a long short-term memory network (LSTM) can be used to process the time series part of the motion capture data, and then these features can be combined with the fully connected layer to integrate these features with the structured data in the disease database (such as disease classification codes, treatment effect indicators, etc.). The output of the model can be a prediction result of the disease state (such as the probability of disease progression, whether complications may occur, etc.) or an evaluation indicator of the physical state (such as physical function score, exercise risk level, etc.).

[0120] Optionally, in step S3, the deep learning model is trained by annotating the data, the parameters of the model are continuously adjusted, the loss function of the model is optimized, and the prediction accuracy of the model is improved. Specifically, the constructed deep learning model is trained using the annotated data, the parameters of the model are continuously adjusted by the back propagation algorithm, the loss function of the model is optimized, and the prediction accuracy of the model is improved. During the training process, cross-validation is used to prevent overfitting to ensure that the deep learning model has good generalization ability for new data.

[0121] The following is a detailed explanation of how to adjust parameters and optimize loss functions in deep learning model training:

[0122] The key steps and main formulas of the deep learning model that integrates image, motion capture and disease data are:

[0123] 1. Model Construction

[0124] 1. Image data processing (CNN)

[0125] Convolutional Layer:

[0126] For the lth convolutional layer, the input feature map X l-1 , convolution kernel W l , bias b l , output feature map X l :

[0127]

[0128] Where K l is the convolution kernel size, C l-1 is the number of input channels, and d represents the output channel index.

[0129] Activation function (such as ReLU): A l =max(0,X l )

[0130] Pooling layer (such as max pooling): s is the pooling step size and R is the pooling window.

[0131] 2. Motion capture data processing (LSTM)

[0132] LSTM Cell:

[0133] For time step t:

[0134] i t =σ(W ii x t +b ii +W hi h t-1 +b hi )

[0135] f t =σ(W if x t +b if +W hf h t-1 +b hf )

[0136] g t =σ(W ig x t +b ig +W hg h t-1 +b hg )

[0137] o t =σ(W io x t +b io +W ho h t-1 +b ho )

[0138] c t =f t ⊙c t-1 +i t ⊙g t

[0139] h t =o t ⊙tanh(c t )

[0140] 3. Feature Fusion

[0141] The processed image features v img , action features act and condition data x cond Splicing: v combined =[v img ,v act ,x cond ]

[0142] 4. Output Layer

[0143] Map the fused features to the output through a fully connected layer:

[0144] y=σ(W out v combind +b out )

[0145] 2. Loss Function

[0146] 1. Classification task (cross entropy)

[0147] For multi-classification:

[0148] Model Output True label y∈{0,1} C :

[0149]

[0150] 2. Regression task (mean square error)

[0151] Model Output The true label y∈R:

[0152]

[0153] 3. Parameter adjustment

[0154] 1. Back Propagation

[0155] Compute the gradient:

[0156] For each layer parameter, the gradient is calculated according to the chain rule.

[0157] For example, for the weights W of the convolutional layer l :

[0158]

[0159] 2. Optimization algorithm (taking Adam as an example)

[0160] Adam Optimizer:

[0161] First moment estimate of the gradient:

[0162]

[0163] Second moment estimate of the gradient:

[0164]

[0165] Bias correction:

[0166]

[0167] Update formula:

[0168]

[0169] 4. Training Process

[0170] 1. Data preprocessing

[0171] Standardize or normalize the data.

[0172] 2. Training steps

[0173] Initialization: Randomly initialize model parameters.

[0174] Divide the data set into: training set, validation set, and test set.

[0175] Training loop:

[0176] Forward propagation: Calculate the prediction results.

[0177] Calculate the loss.

[0178] Backpropagation: Calculate gradients.

[0179] Parameter Update: Update the parameters using an optimization algorithm.

[0180] 5. Cross-Validation

[0181] K-fold cross validation:

[0182] Divide the dataset into K parts, use K-1 parts for training and 1 part for validation, and repeat K times.

[0183] Specifically, a large amount of existing annotated data is used to train the constructed multimodal fusion model. The annotated data includes historical patient images, motion capture data, and corresponding disease progression results and physical condition assessment results. The model parameters are continuously adjusted through the back-propagation algorithm to optimize the model's loss function and improve the model's prediction accuracy. During the training process, cross-validation and other techniques are used to prevent overfitting and ensure that the model has good generalization capabilities for new data. For example, the data set is divided into training set, validation set, and test set, and the model performance is regularly evaluated on the validation set. When the model's performance on the validation set no longer improves, the training is stopped, and the model is finally evaluated and adjusted based on the results of the test set.

[0184] Optionally, in step S4, identifying and extracting key entities from the data processed by the deep learning model, determining the relationship between the extracted key entities, constructing a knowledge graph system, and updating and optimizing the knowledge graph system include the following steps:

[0185] S401. Identify and extract key entities from the data processed by the deep learning model. The key entities include disease name, body part, symptom manifestation, treatment method, movement characteristics, and extract historical data of the subject from the disease database.

[0186] Specifically, key entities are identified and extracted from the data processed by the deep learning model, including disease names, body parts, symptoms, treatment methods, motion characteristics, etc. For example, specific abnormal body postures identified from images may correspond to symptoms of a certain disease, and abnormal motion patterns extracted from motion capture data can also be used as an entity. At the same time, basic information about patients, diagnosis results, treatment processes, and other entity information are extracted from the disease database.

[0187] S402. Extract the relationship between key entities and historical data and build a knowledge graph system.

[0188] Specifically, determine the relationship between the extracted entities and construct the semantic network of the knowledge graph. For example, there is a causal relationship between a certain disease and abnormal movement of a specific body part, and there is a correlation between a certain treatment method and the improvement effect of the disease. Through these relationships, different entities are connected to form a knowledge graph with a directed graph structure. For example, there is a "cause" relationship between "Parkinson's disease" and "hand tremor", and there is a "treatment" relationship between "drug treatment A" and "Parkinson's disease".

[0189] S403. Add the relationship between the newly discovered key entities and historical data to the knowledge graph system, and modify and improve the existing relationships.

[0190] As new data is continuously collected and models are continuously trained, the knowledge graph needs to be continuously updated and optimized. Newly discovered entities and relationships are added to the knowledge graph, and existing relationships are corrected and improved. For example, if a new movement feature is found to be associated with a specific disease, it is added to the knowledge graph; if a previously established relationship between a treatment method and the improvement of a disease is found to be deviated, adjustments are made based on new data.

[0191] In step S5, the first data is integrated with the historical data in the disease database and input into the trained deep learning model and knowledge graph system to perform disease prediction and physical condition monitoring on the subject, as follows:

[0192] The images and motion capture data collected in real time during the rehabilitation training are integrated with the historical data in the disease database and input into the trained deep learning model and knowledge graph system. The model performs disease prediction analysis based on the input data, such as predicting the risk of disease recurrence, the direction and speed of disease progression, etc. For example, if the model finds that the patient's motor ability continues to decline and some new abnormal movement patterns appear, combined with the relationship between the disease and movement characteristics in the knowledge graph, it is predicted that the disease may be about to enter a new stage of development and the treatment plan needs to be adjusted in time.

[0193] Prediction result visualization and early warning: disease prediction results are presented to medical professionals and patients in an intuitive way. For example, the probability curve of disease progression and the risk level of possible complications are displayed through a visual interface. When the prediction results reach a certain early warning threshold, the system automatically issues an early warning message to remind medical staff to pay attention to changes in the patient's condition and take appropriate measures. For example, if it is predicted that the patient has a higher risk of cardiovascular disease complications in the next week, the system sends a warning message to the doctor and provides relevant data support and recommended examination items.

[0194] Dynamic evaluation and trend analysis: Use deep learning models and knowledge graphs to continuously monitor and dynamically evaluate the physical condition of the subjects. The model calculates physical function scores, motor coordination indicators, fatigue levels and other physical condition parameters based on real-time collected data, and compares and analyzes them with historical data to derive the changing trend of physical condition. For example, if it is found that the patient's motor coordination gradually improves over a period of time, it means that rehabilitation training has achieved certain results; if the physical function score suddenly drops, it may indicate that the patient has new health problems or excessive rehabilitation training.

[0195] Personalized rehabilitation advice and adjustments: Based on the results of physical condition monitoring, provide subjects with personalized rehabilitation advice and training plan adjustment plans. For example, if a patient is found to have poor performance in a specific movement, combined with the relationship between the movement and related muscle groups and body parts in the knowledge graph, recommend targeted rehabilitation training programs; if the patient is highly fatigued, it is recommended to appropriately reduce the training intensity or increase the rest time. In this way, the rehabilitation treatment can be personalized and precise, improving the rehabilitation effect and the patient's quality of life.

[0196] Optionally, in step S105, joint velocity features are extracted from the motion capture data using a first formula, which is:

[0197]

[0198] (1) where Δt is the time interval, θ(t) is the joint angle, t is the time, and v(t) is the joint velocity.

[0199] The joint acceleration is calculated for the motion capture data using the second formula, which is:

[0200]

[0201] (2)In the formula, a(t) is the joint acceleration.

[0202] For example, if we collect data at 100 frames per second, Δt = 0.01 seconds, when θ(t) = 0.5 radians, θ(t-0.01) = 0.4 radians,

[0203] Joint acceleration calculation:

[0204] Optionally, the time domain feature of heart rate variability is extracted from the physiological data using a third formula, and the third formula is:

[0205]

[0206] (3) where N is the heartbeat interval within a period of time, RR n is the heartbeat interval sequence, n is a positive integer less than or equal to N, is the mean value of the heartbeat interval sequence, and SDNN is the standard deviation.

[0207] For example, there is a sequence RR n ={900, 920, 950, 880, 930 (unit: milliseconds),

[0208] Calculate first Then calculate

[0209]

[0210] Optionally, muscle electrical signal features are extracted from physiological data using a fourth formula, which is:

[0211]

[0212] (4) In the formula, x(n) is the muscle electrical signal sequence, n = 0, 1, ..., N-1, and x(n) is transformed by FFT to obtain X(k).

[0213] For example, for a simple muscle electrical signal sequence x(n)=sin2πf0n (f0 is the signal frequency), after FFT transformation, the corresponding power spectrum density can be calculated according to the above formula to analyze the frequency component of the signal.

[0214] The above is only the best specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for constructing a rehabilitation platform, characterized in that: The following steps are involved: Provide a device or place for safety support, conduct limb movement, coordination, balance or walking training for the subject, collect data from the subject through images, motion capture equipment, wearable detection equipment and environmental sensor equipment to obtain first data, pre-process and feature extract the first data in sequence to obtain second data, connect the first data with the disease database, and determine the mapping relationship between the first data and the historical data in the disease database; A deep learning model is constructed, the second data is used as input to the deep learning model, and third data is output, wherein the third data includes the evaluation results of the subject's limb function status, the prediction of rehabilitation training effects, and the identification of abnormal movement patterns.

2. The method for constructing a rehabilitation platform according to claim 1, characterized in that: The step of constructing the deep learning model, taking the second data as input to the deep learning model, and outputting the third data further includes: The deep learning model is trained by annotating data, the parameters of the model are continuously adjusted, the loss function of the model is optimized, and the prediction accuracy of the model is improved. The annotated data includes the evaluation results of the limb function status of the subject by a professional rehabilitation therapist, the rehabilitation training effect, and the actual observation data; Identify and extract key entities from the data processed by the deep learning model, determine the relationships between the extracted key entities, build a knowledge graph system, and update and optimize the knowledge graph system; The first data is integrated with the historical data in the disease database and input into the trained deep learning model and the knowledge graph system to perform disease prediction and physical condition monitoring on the subjects.

3. The method for constructing a rehabilitation platform according to claim 1, characterized in that: The step of constructing the deep learning model, taking the second data as input to the deep learning model, and outputting the third data further includes: The third data is connected to the U3D / UE4 development engine in real time, and the real-time driving and rendering technology of the virtual human is used to present the real-time motion state on the display screen, so as to achieve a high degree of matching of the real-time driving of the virtual human; A virtual online platform is constructed, and subjects enter the platform as virtual people to conduct three-dimensional interaction, which includes online medical consultation and online treatment.

4. The method for constructing a rehabilitation platform according to claim 1, characterized in that: The step of sequentially preprocessing and feature extracting the first data comprises the following steps: Cleaning the image data in the first data, removing interference factors, and performing image standardization processing to ensure the consistency of image data from different sources; Converting and calibrating the motion capture data obtained by the motion capture device in the first data into a format suitable for processing by a deep learning framework, and correcting the motion capture data according to the calibration parameters of the device to improve the accuracy of the data; Filtering the physiological data recorded by the wearable monitoring device in the first data to remove abnormal values ​​and baseline drift to make the data smoother and more stable, wherein the physiological data includes heart rate, blood pressure, and muscle electrical signals; Using a convolutional neural network to extract features from the image data; Extracting joint angle, joint velocity, joint acceleration and motion time features from the motion capture data; The time domain characteristics of heart rate variability, exercise intensity and muscle electrical signal characteristics are extracted from the physiological data.

5. The method for constructing a rehabilitation platform according to claim 1, characterized in that: The historical data includes the patient's case information, diagnosis results, and treatment process records; Determine the mapping relationship between the first data and the historical data in the disease database, specifically: use the subject's medical record number or unique identifier as an association key, and match the first data with the corresponding medical record information.

6. The method for constructing a rehabilitation platform according to claim 1, characterized in that: The construction of the deep learning model is specifically as follows: The second data is processed using a convolutional neural network, a recurrent neural network or a long short-term memory network, and then the features of the second data in the fully connected layer are fused with the structured data in the disease database.

7. The method for constructing a rehabilitation platform according to claim 2, characterized in that: The deep learning model is trained by annotating data, the parameters of the model are continuously adjusted, the loss function of the model is optimized, and the prediction accuracy of the model is improved, specifically: The deep learning model constructed is trained using the labeled data, and the parameters of the model are continuously adjusted through the back propagation algorithm to optimize the loss function of the model and improve the prediction accuracy of the model. During the training process, cross-validation is used to prevent overfitting and ensure that the deep learning model has good generalization ability for new data.

8. The method for constructing a rehabilitation platform according to claim 2 or 7, characterized in that: The step of identifying and extracting key entities from the data processed by the deep learning model, determining the relationships between the extracted key entities, constructing a knowledge graph system, and updating and optimizing the knowledge graph system includes the following steps: Identify and extract key entities from the data processed by the deep learning model, the key entities including disease name, body part, symptom manifestation, treatment method, movement characteristics, and extract historical data of the subject from the disease database; Extracting the relationship between the key entities and the historical data to construct a knowledge graph system; The relationship between the newly discovered key entities and the historical data is added to the knowledge graph system, and the existing relationships are corrected and improved.

9. The method for constructing a rehabilitation platform according to claim 8, characterized in that: The joint velocity feature is extracted from the motion capture data using a first formula, where the first formula is: Where Δt is the time interval, θ(t) is the joint angle, t is the time, and v(t) is the joint velocity; The joint acceleration is calculated for the motion capture data using a second formula, where the second formula is: Where a(t) is the joint acceleration; The time domain feature extraction of heart rate variability is performed on the physiological data by using the third formula, and the third formula is: Among them, N is the heartbeat interval within a period of time, RR n is the heartbeat interval sequence, n is a positive integer less than or equal to N, is the mean value of the heartbeat interval sequence, and SDNN is the standard deviation.

10. The method for constructing a rehabilitation platform according to claim 9, characterized in that: The muscle electrical signal feature extraction is performed on the physiological data using the fourth formula, and the fourth formula is: Wherein, x(n) is a muscle electrical signal sequence, n=0, 1, ..., N-1, and x(n) is transformed by FFT to obtain X(k).

Images