Postoperative bent leg training monitoring system and method
By extracting keyframes of the gait cycle from the bending leg training monitoring video and using convolutional neural network and ViT model, the problem of gait abnormality assessment of postoperative bending leg patients is solved, and the accurate evaluation and timely processing of gait is achieved. , ensure the smooth progress of postoperative rehabilitation.
Patent Information
- Application Number
- CN202310530445.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-05-12
AI Technical Summary
Postoperative patients with bent legs may experience knee flexion when walking, resulting in abnormal gait and affecting postoperative rehabilitation. It is difficult for the existing technology to effectively evaluate and detect such abnormal conditions.
Keyframes within the gait period are extracted from the bending leg training monitoring video, gait energy maps are generated, and the first and second scale gait energy feature maps are extracted using convolutional neural network and ViT model. After fusion, the classifier is judged Is the gait normal?
Accurate assessment of the patient's gait is achieved, timely detection of abnormalities, and ensuring appropriate treatment during postoperative rehabilitation.
Smart Images

Figure CN116631619B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent monitoring, and more specifically, to a post-operative bent leg training monitoring system and method. Background Art
[0002] Functional exercise is a necessary means to ensure joint function, restore muscle strength, and accelerate the recovery process. For patients who do bent-leg exercise after surgery, appropriate activities can promote blood circulation and reduce the risk of thrombosis. However, patients with bent-leg exercise after surgery may experience knee flexion when walking, resulting in abnormal gait and affecting the patient's postoperative recovery.
[0003] Therefore, an optimized postoperative bent-leg training monitoring system is desired, which can evaluate the patient's gait and detect abnormal conditions in time so as to take corresponding measures. Summary of the invention
[0004] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a postoperative bent leg training monitoring system and method. It first extracts multiple bent leg training monitoring key frames within a gait cycle from the bent leg training monitoring video of the monitored object, then processes the multiple bent leg training monitoring key frames to obtain a gait energy map, then passes the gait energy map through a convolutional neural network model and a ViT model respectively to obtain a first-scale gait energy feature map and a second-scale gait energy feature map, then, fuses the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map, and finally, passes the classification feature map through a classifier to obtain a classification result for indicating whether the gait of the monitored object is normal. In this way, the patient's gait assessment can be accurately performed, and abnormalities can be discovered in time to facilitate corresponding processing.
[0005] According to one aspect of the present application, a postoperative bent leg training monitoring system is provided, comprising:
[0006] A video data acquisition module, used for acquiring a monitoring video of a bent leg training of a subject to be monitored;
[0007] A key frame extraction module, used for extracting a plurality of bent-leg training monitoring key frames within a gait cycle from the bent-leg training monitoring video of the subject to be monitored;
[0008] A gait energy graph generating module, used for processing the plurality of bent-leg training monitoring key frames to obtain a gait energy graph;
[0009] A first scale image feature extraction module, used for passing the gait energy graph through a convolutional neural network model as a feature extractor to obtain a first scale gait energy feature graph;
[0010] A second-scale image feature extraction module, used for passing the gait energy map through the ViT model to obtain a second-scale gait energy feature map;
[0011] a multi-scale image feature association module, configured to fuse the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and
[0012] The gait detection module is used to pass the classification feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether the gait of the monitored object is normal.
[0013] In the above-mentioned post-operative bent leg training monitoring system, the first scale image feature extraction module is used to:
[0014] Each layer of the convolutional neural network model as a feature extractor is performed separately in the forward pass of the layer:
[0015] Performing convolution processing on the input data to obtain a first-scale convolution feature map;
[0016] Performing mean pooling processing on the first-scale convolutional feature map to obtain a first-scale pooled feature map; and
[0017] Performing nonlinear activation on the first-scale pooling feature map to obtain a first-scale activation feature map;
[0018] Among them, the output of the last layer of the convolutional neural network model serving as a feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as a feature extractor is the gait energy map.
[0019] In the above-mentioned postoperative bent leg training monitoring system, the second scale image feature extraction module includes:
[0020] An image block unit, used for performing image block processing on the gait energy map to obtain a sequence of image blocks;
[0021] An embedding coding unit, configured to use the embedding layer of the ViT model to perform embedding coding on each image block in the sequence of image blocks to obtain a sequence of image block embedding vectors;
[0022] An image context semantic association unit, configured to input the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and
[0023] A dimensionality reconstruction unit is used to arrange the sequence of the image block context semantic feature vectors according to the corresponding spatial positions during the image block processing to obtain the second-scale gait energy feature map.
[0024] In the above-mentioned postoperative bent leg training monitoring system, the embedded coding unit is used to:
[0025] Arranging pixel values of all pixel positions in each image block in the sequence of image blocks into a one-dimensional vector; and
[0026] The one-dimensional vector is fully connected encoded using the fully connected layer of the ViT model using the following fully connected encoding formula to extract high-dimensional implicit features of the eigenvalues at each position in the one-dimensional vector, wherein the fully connected encoding formula is: Where X is the one-dimensional vector, Y is the output vector, W is the weight matrix, and B is the bias vector. Represents matrix multiplication.
[0027] In the above-mentioned postoperative bent leg training monitoring system, the multi-scale image feature association module is used to:
[0028] Performing scale-aware Gaussian joint density fusion on the first-scale gait energy feature map and the second-scale gait energy feature map using the following optimization formula to obtain the classification feature map;
[0029] Wherein, the optimization formula is:
[0030]
[0031]
[0032] f1∈F1 and f2∈F2
[0033] Among them, F1 represents the first-scale gait energy feature map, F2 represents the second-scale gait energy feature map, f1 is the feature value of each position in the first-scale gait energy feature map, f2 is the feature value of each position in the second-scale gait energy feature map, μ and σ are the mean and variance of the feature set of all feature values of the first-scale gait energy feature map and the second-scale gait energy feature map, respectively, W, H and C are the width, height and number of channels of the feature map, respectively, and f′ is the feature value of each position of the classification feature map.
[0034] In the above-mentioned postoperative bent leg training monitoring system, the gait detection module includes:
[0035] A feature map expansion unit, used to expand the classification feature map into a classification feature vector according to a row vector or a column vector;
[0036] a fully connected encoding unit, configured to perform fully connected encoding on the classification feature vector using the fully connected layer of the classifier to obtain an encoded classification feature vector; and
[0037] The classification unit is used to input the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.
[0038] According to another aspect of the present application, a postoperative bent leg training monitoring method is provided, comprising:
[0039] Obtaining a monitoring video of a bent-leg training of a subject to be monitored;
[0040] Extracting a plurality of bent-leg training monitoring key frames within a gait cycle from the bent-leg training monitoring video of the subject to be monitored;
[0041] Processing the plurality of bent-leg training monitoring key frames to obtain a gait energy graph;
[0042] Passing the gait energy graph through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature graph;
[0043] Passing the gait energy graph through the ViT model to obtain a second-scale gait energy feature graph;
[0044] fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and
[0045] The classification feature map is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the gait of the monitored object is normal.
[0046] In the above-mentioned postoperative bent leg training monitoring method, the gait energy graph is passed through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature graph, including:
[0047] Each layer of the convolutional neural network model as a feature extractor is performed separately in the forward pass of the layer:
[0048] Performing convolution processing on the input data to obtain a first-scale convolution feature map;
[0049] Performing mean pooling processing on the first-scale convolutional feature map to obtain a first-scale pooled feature map; and
[0050] Performing nonlinear activation on the first-scale pooling feature map to obtain a first-scale activation feature map;
[0051] Among them, the output of the last layer of the convolutional neural network model serving as a feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as a feature extractor is the gait energy map.
[0052] In the above-mentioned postoperative bent leg training monitoring method, the gait energy graph is passed through the ViT model to obtain a second-scale gait energy characteristic graph, including:
[0053] Performing image block processing on the gait energy map to obtain a sequence of image blocks;
[0054] Using the embedding layer of the ViT model, embedding coding is performed on each image block in the sequence of image blocks to obtain a sequence of image block embedding vectors;
[0055] Inputting the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block contextual semantic feature vectors; and
[0056] The sequence of the image block context semantic feature vectors is arranged according to the corresponding spatial positions during the image block processing to obtain the second-scale gait energy feature map.
[0057] In the above-mentioned postoperative bent leg training monitoring method, each image block in the sequence of image blocks is embedded and encoded using the embedding layer of the ViT model to obtain a sequence of image block embedding vectors, including:
[0058] Arranging pixel values of all pixel positions in each image block in the sequence of image blocks into a one-dimensional vector; and
[0059] The one-dimensional vector is fully connected encoded using the fully connected layer of the ViT model using the following fully connected encoding formula to extract high-dimensional implicit features of the eigenvalues at each position in the one-dimensional vector, wherein the fully connected encoding formula is: Where X is the one-dimensional vector, Y is the output vector, W is the weight matrix, and B is the bias vector. Represents matrix multiplication.
[0060] Compared with the prior art, the postoperative bent leg training monitoring system and method provided by the present application first extracts multiple bent leg training monitoring key frames within a gait cycle from the bent leg training monitoring video of the subject to be monitored, then processes the multiple bent leg training monitoring key frames to obtain a gait energy map, then passes the gait energy map through a convolutional neural network model and a ViT model respectively to obtain a first-scale gait energy feature map and a second-scale gait energy feature map, then fuses the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map, and finally passes the classification feature map through a classifier to obtain a classification result for indicating whether the gait of the subject to be monitored is normal. In this way, the patient's gait assessment can be accurately performed, and abnormalities can be discovered in time to facilitate corresponding processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The following drawings are not deliberately scaled according to the actual size, and the focus is on illustrating the main purpose of the present application.
[0062] Figure 1 This is an application scenario diagram of the postoperative bent leg training monitoring system according to an embodiment of the present application.
[0063] Figure 2 It is a block diagram of a postoperative bent leg training monitoring system according to an embodiment of the present application.
[0064] Figure 3 Schematic diagram of the block diagram of the second-scale image feature extraction module in the post-operative leg bending training monitoring system according to an embodiment of the present application.
[0065] Figure 4 It is a block diagram of the gait detection module in the postoperative bent leg training monitoring system according to an embodiment of the present application.
[0066] Figure 5 Flow chart of a method for monitoring postoperative bent leg training according to an embodiment of the present application.
[0067] Figure 6 Schematic diagram of the system architecture of the postoperative bent leg training monitoring method according to an embodiment of the present application. DETAILED DESCRIPTION
[0068] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.
[0069] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.
[0070] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.
[0071] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.
[0072] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0073] As mentioned above, the following monitoring items need to be paid attention to in postoperative bow leg monitoring: Activity status. For patients with bow legs after surgery, appropriate activities can promote blood circulation and reduce the risk of thrombosis. Therefore, it is necessary to pay close attention to the patient's activity status and perform appropriate activities according to the doctor's instructions; Pain assessment. After surgery, patients may feel pain and discomfort. It is necessary to assess the patient's pain and treat it according to the doctor's medication recommendations; Gait assessment. Patients with bow legs after surgery may have knee flexion when walking, resulting in abnormal gait. It is necessary to assess the patient's gait and detect abnormal conditions in time so that corresponding treatment can be carried out; Lower limb activity assessment. Monitor the patient's lower limb activity, which can help monitor muscle traction and joint range, increase lower limb circulation and reduce the risk of deep vein thrombosis in the limbs. The above monitoring items require professionals to scientifically monitor patients, detect abnormal conditions in time, and take appropriate treatment measures in time to ensure that patients can recover better.
[0074] Furthermore, for patients who perform bent-leg exercises after surgery, appropriate activities can promote blood circulation and reduce the risk of thrombosis. However, patients with bent-leg exercises after surgery may experience knee flexion when walking, resulting in abnormal gait and affecting the patient's postoperative recovery. Therefore, an optimized postoperative bent-leg training monitoring system is desired, which can evaluate the patient's gait and detect abnormalities in time so that corresponding treatment can be carried out.
[0075] Accordingly, considering that the patient is undergoing postoperative bent leg training, in order to be able to evaluate the patient's gait and find abnormalities in time, it is expected to be achieved by analyzing the bent leg training monitoring video of the monitored object. That is to say, the bent leg training monitoring video of the monitored object contains information about the gait characteristics of the patient during the postoperative bent leg training. Therefore, the gait monitoring of the patient during the postoperative bent leg training is achieved based on the analysis of the video. However, since there is a large amount of information in the bent leg training monitoring video, the gait characteristic information about the patient's postoperative bent leg training is a small-scale hidden feature in the monitoring video, and it is difficult to fully express the features. Therefore, in this process, the difficulty lies in how to fully dig out the implicit feature distribution information about the patient's postoperative bent leg training gait in the bent leg training monitoring video, so as to accurately evaluate the patient's gait and find abnormalities in time to facilitate corresponding processing.
[0076] In recent years, deep learning and neural networks have been widely used in computer vision, natural language processing, text signal processing and other fields. The development of deep learning and neural networks provides new solutions and solutions for mining the implicit feature distribution information of the patient's postoperative bent leg training gait in the bent leg training monitoring video.
[0077] Specifically, in the technical solution of the present application, first, a monitoring video of bent leg training of the subject to be monitored is obtained. Next, considering that in the bent leg training monitoring video, the gait change characteristics of the subject to be monitored can be represented by the difference between adjacent monitoring frames within a gait cycle in the bent leg training monitoring video, that is, the gait changes of the patient during the bent leg training after surgery are represented by the image representation of adjacent image frames. However, considering that the difference between adjacent frames in the monitoring video is small and there is a large amount of data redundancy, therefore, in order to reduce the amount of calculation and avoid the adverse effects of data redundancy on detection, key frame sampling is performed on the bent leg training monitoring video at a predetermined sampling frequency to extract multiple bent leg training monitoring key frames within a gait cycle from the bent leg training monitoring video of the subject to be monitored.
[0078] Then, considering that the gait energy image (GEI) is a very commonly used gait feature in gait detection, it can well express the characteristic information of the gait such as speed and morphology. Therefore, in the technical solution of the present application, the multiple bent-leg training monitoring key frames are processed to obtain a gait energy image. In particular, in a specific example of the present application, the generation of the gait energy image mainly includes: on the original contour image, that is, the contour of the patient in each bent-leg training monitoring key frame is cropped, and the top of the patient's head is used as the center position; then, the cropped image is synthesized to synthesize a gait energy image with the image of a gait cycle.
[0079] Furthermore, a convolutional neural network model as a feature extractor with excellent performance in extracting implicit features of images is used to perform feature mining of the gait energy map, so as to extract implicit feature distribution information such as gait speed and morphology of the patient during postoperative bent leg training, thereby obtaining a first-scale gait energy feature map.
[0080] Then, considering that due to the inherent limitations of convolution operations, it is difficult for pure CNN methods to learn clear global and long-range semantic information interactions. In addition, it is also considered that the implicit features of the patient's gait speed, morphology, etc. in the gait energy map during the postoperative bent leg training are small-scale subtle features, which are difficult to capture and extract. Therefore, in order to improve the expression ability of the patient's gait implicitly contained in the gait energy map, so as to improve the accurate evaluation of the patient's gait, in the technical solution of the present application, the gait energy map is encoded through the ViT model to extract the implicit global contextual semantic association feature distribution information of the patient's gait speed and morphology during the postoperative bent leg training in the gait energy map, thereby obtaining a second-scale gait energy feature map. It should be understood that the small-scale implicit features of the patient's gait speed and morphology in each image block after the image block processing of the gait energy map are no longer small-scale feature information, which is conducive to the subsequent detection of whether the patient's gait is normal. In particular, here, the embedding layer linearly projects each image block into a one-dimensional embedding vector through a learnable embedding matrix. The embedding implementation process is to first arrange the pixel values of all pixel positions in each image block into a one-dimensional vector, and then use a fully connected layer to fully connect the one-dimensional vector to achieve embedding. Moreover, here, the ViT model can directly process each image block through a self-attention mechanism like a Transformer, so as to extract the global implicit contextual semantic association feature information about the patient's gait speed and morphology during postoperative bent leg training based on the overall gait energy map in each image block.
[0081] Then, the first scale gait energy feature map and the second scale gait energy feature map are further fused to obtain a classification feature map, so as to fuse the multi-scale correlation feature distribution information based on long-distance dependency association and short-distance dependency association about the implicit features of the patient's gait speed, morphology, etc. in the bent leg training monitoring video during the training process. Then, the classification feature map is classified in a classifier to obtain a classification result indicating whether the gait of the monitored object is normal.
[0082] That is, in the technical solution of the present application, the label of the classifier includes the gait of the object to be monitored being normal (first label), and the gait of the object to be monitored being abnormal (second label), wherein the classifier determines which classification label the classification feature map belongs to by means of a soft maximum function. It is worth noting that the first label p1 and the second label p2 here do not contain artificially set concepts. In fact, during the training process, the computer model does not have the concept of "whether the gait of the object to be monitored is normal". It only has two classification labels and the probability of the output feature under these two classification labels, that is, the sum of p1 and p2 is one. Therefore, the classification result of whether the gait of the object to be monitored is normal is actually converted into a binary class probability distribution that conforms to natural laws through the classification label. In essence, the physical meaning of the natural probability distribution of the label is used, rather than the linguistic text meaning of "whether the gait of the object to be monitored is normal". It should be understood that in the technical solution of the present application, the classification label of the classifier is an evaluation detection label for whether the gait of the monitored object is normal. Therefore, after obtaining the classification result, the patient's gait during postoperative bent leg training can be accurately evaluated based on the classification result, so as to detect abnormalities in time to facilitate corresponding processing.
[0083] In particular, in the technical solution of the present application, when the first-scale gait energy feature map and the second-scale gait energy feature map are fused to obtain the classification feature map, since the first-scale gait energy feature map and the second-scale gait energy feature map respectively express the image semantics of the gait energy maps at different scales, that is, the first-scale gait energy feature map expresses the local correlation image semantics at the convolution kernel scale of the feature extractor, and the second-scale gait energy feature map expresses the contextual image semantics at the image block scale of the ViT model, it is expected to fuse the first-scale gait energy feature map and the second-scale gait energy feature map at different scales to enhance the fusion effect of the classification feature map on the first-scale gait energy feature map and the second-scale gait energy feature map.
[0084] Based on this, the applicant of the present application performs Gaussian joint density fusion based on scale cognition on the first scale gait energy feature graph F1 and the second scale gait energy feature graph F2, which is expressed as:
[0085]
[0086]
[0087] f1∈F1 and f2∈F2
[0088] μ and σ are The mean and variance of the feature set (i.e., the feature set of all eigenvalues of the first-scale gait energy feature map F1 and the second-scale gait energy feature map F2), W, H and C are the width, height and number of channels of the feature map, respectively, and f′ is the eigenvalue of the classification feature map.
[0089] Here, the scale-aware Gaussian joint density fusion takes into account the scale-optimal expression characteristics of the features to be fused. In order to improve the effectiveness of feature fusion and the generalization performance relative to the features to be fused, the scale-aware Gaussian joint density is used as an advantage function to strategically represent the classification performance gap of feature distribution based on mean and variance, thereby improving the feature-scale self-dependence of feature fusion, so as to improve the fusion effect of the classification feature map on the first-scale gait energy feature map and the second-scale gait energy feature map. In this way, the gait detection and evaluation of patients during postoperative bent leg training can be accurately carried out, and abnormalities can be found in time to facilitate corresponding treatment.
[0090] Figure 1 FIG. 1 is an application scenario diagram of the postoperative leg bending training monitoring system according to an embodiment of the present application. Figure 1 As shown, in this application scenario, first, a surveillance camera (e.g., Figure 1 The device C) shown in the figure obtains the bent leg training monitoring video of the monitored object (for example, Figure 1 D), then, the bent leg training monitoring video of the subject to be monitored is input into a server deployed with a postoperative bent leg training monitoring algorithm (for example, Figure 1 S) as shown in , wherein the server is able to use the post-operative bent leg training monitoring algorithm to process the bent leg training monitoring video of the monitored object to obtain a classification result indicating whether the gait of the monitored object is normal.
[0091] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.
[0092] Figure 2 FIG. 1 is a block diagram of a postoperative leg bending training monitoring system according to an embodiment of the present application. Figure 2As shown, according to the embodiment of the present application, the postoperative bent leg training monitoring system 100 includes: a video data acquisition module 110, which is used to obtain the bent leg training monitoring video of the monitored object; a key frame extraction module 120, which is used to extract multiple bent leg training monitoring key frames within a gait cycle from the bent leg training monitoring video of the monitored object; a gait energy map generation module 130, which is used to process the multiple bent leg training monitoring key frames to obtain a gait energy map; a first-scale image feature extraction module 140, which is used to pass the gait energy map through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature map; a second-scale image feature extraction module 150, which is used to pass the gait energy map through a ViT model to obtain a second-scale gait energy feature map; a multi-scale image feature association module 160, which is used to fuse the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and a gait detection module 170, which is used to pass the classification feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether the gait of the monitored object is normal. Therefore, in order to be able to evaluate the patient's gait and find abnormalities in time, it is expected to be achieved by analyzing the monitoring video of the bent leg training of the monitored object. That is to say, the bent leg training monitoring video of the monitored object contains information about the gait characteristics of the patient during the postoperative bent leg training. Therefore, the gait monitoring of the patient during the postoperative bent leg training is achieved based on the analysis of the video. However, since there is a large amount of information in the bent leg training monitoring video, the gait characteristic information about the patient's postoperative bent leg training is a small-scale hidden feature in the monitoring video, and it is difficult to fully express the features. Therefore, in the technical solution of the present application, the implicit feature distribution information about the patient's postoperative bent leg training gait in the bent leg training monitoring video is mined to accurately evaluate the patient's gait, and to find abnormalities in time to facilitate corresponding processing.
[0093] More specifically, in the embodiment of the present application, the video data acquisition module 110 is used to obtain the monitoring video of the bent leg training of the monitored object. For patients who do bent leg training after surgery, appropriate activities can promote blood circulation and reduce the risk of thrombosis. However, patients with bent legs after surgery may experience knee flexion when walking, resulting in abnormal gait and affecting the patient's postoperative recovery.
[0094] More specifically, in an embodiment of the present application, the key frame extraction module 120 is used to extract multiple bent leg training monitoring key frames within a gait cycle from the bent leg training monitoring video of the subject to be monitored. In the bent leg training monitoring video, the gait change characteristics of the subject to be monitored can be represented by the difference between adjacent monitoring frames within a gait cycle in the bent leg training monitoring video, that is, the gait changes of the patient during the bent leg training after surgery are represented by the image representation of adjacent image frames. However, considering that the difference between adjacent frames in the monitoring video is small and there is a large amount of data redundancy, therefore, in order to reduce the amount of calculation and avoid the adverse effects of data redundancy on detection, key frame sampling is performed on the bent leg training monitoring video at a predetermined sampling frequency to extract multiple bent leg training monitoring key frames within a gait cycle from the bent leg training monitoring video of the subject to be monitored.
[0095] More specifically, in an embodiment of the present application, the gait energy map generation module 130 is used to process the multiple bent-leg training monitoring key frames to obtain a gait energy map. The gait energy map is a very commonly used gait feature in gait detection, which can well express the characteristic information such as the speed and shape of the gait. Therefore, in the technical solution of the present application, the multiple bent-leg training monitoring key frames are processed to obtain a gait energy map. In particular, in a specific example of the present application, the gait energy map generation mainly includes: on the original contour map, that is, the contour of the patient in each bent-leg training monitoring key frame is cropped, and the top of the patient's head is used as the center position; then, the cropped image is synthesized to synthesize a gait energy map with the image of a gait cycle.
[0096] More specifically, in the embodiment of the present application, the first-scale image feature extraction module 140 is used to pass the gait energy graph through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature graph. The convolutional neural network model as a feature extractor with excellent performance in implicit feature extraction of images is used to perform feature mining on the gait energy graph to extract implicit feature distribution information such as gait speed and morphology of the patient during postoperative bent leg training, thereby obtaining a first-scale gait energy feature graph.
[0097] More specifically, in an embodiment of the present application, the first-scale image feature extraction module 140 is used to: each layer of the convolutional neural network model serving as a feature extractor performs in the forward pass of the layer: convolution processing on the input data to obtain a first-scale convolutional feature map; mean pooling processing on the first-scale convolutional feature map to obtain a first-scale pooling feature map; and, nonlinear activation of the first-scale pooling feature map to obtain a first-scale activation feature map; wherein the output of the last layer of the convolutional neural network model serving as a feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as a feature extractor is the gait energy map.
[0098] It should be understood that a convolutional neural network (CNN) is an artificial neural network that is widely used in fields such as image recognition. A convolutional neural network may include an input layer, a hidden layer, and an output layer, wherein the hidden layer may include a convolution layer, a pooling layer, an activation layer, and a fully connected layer, etc. The upper layer performs corresponding operations based on the input data and outputs the operation results to the lower layer. The input initial data is processed through multiple layers of operations to obtain a final result. The convolutional neural network model uses the convolution kernel as a feature filter factor and has excellent performance in extracting local features of images. Compared with traditional image feature extraction algorithms based on statistics or feature engineering, the convolutional neural network model has stronger feature extraction generalization and fitting capabilities.
[0099] More specifically, in an embodiment of the present application, the second-scale image feature extraction module 150 is used to pass the gait energy map through the ViT model to obtain a second-scale gait energy feature map. Considering the inherent limitations of convolution operations, it is difficult for pure CNN methods to learn clear global and long-range semantic information interactions. In addition, it is also considered that the implicit features of the gait energy map about the patient's gait speed, morphology, etc. during the postoperative bent leg training are small-scale subtle features, which are difficult to capture and extract. Therefore, in order to improve the expression ability of the patient's gait implicitly contained in the gait energy map, so as to improve the accurate evaluation of the patient's gait, in the technical solution of the present application, the gait energy map is encoded through the ViT model to extract the implicit global contextual semantic association feature distribution information about the patient's gait speed and morphology during the postoperative bent leg training in the gait energy map, thereby obtaining a second-scale gait energy feature map.
[0100] It should be understood that the small-scale implicit features of the patient's gait speed and morphology during the postoperative bent leg training in each image block after the image block processing of the gait energy map are no longer small-scale feature information, which is conducive to the subsequent detection of whether the patient's gait is normal. In particular, here, the embedding layer linearly projects each image block into a one-dimensional embedding vector through a learnable embedding matrix. The implementation process of embedding is to first arrange the pixel values of all pixel positions in each image block into a one-dimensional vector, and then use a fully connected layer to fully connect the one-dimensional vector to achieve embedding. Moreover, here, the ViT model can directly process each image block through a self-attention mechanism like a Transformer, so as to extract the global implicit contextual semantic association feature information about the patient's gait speed and morphology during the postoperative bent leg training based on the overall gait energy map in each image block.
[0101] More specifically, in the embodiments of the present application, Figure 3 As shown, the second-scale image feature extraction module 150 includes: an image segmentation unit 151, which is used to perform image segmentation processing on the gait energy map to obtain a sequence of image blocks; an embedding coding unit 152, which is used to use the embedding layer of the ViT model to embed each image block in the sequence of image blocks to obtain a sequence of image block embedding vectors; an image context semantic association unit 153, which is used to input the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and a dimensionality reconstruction unit 154, which is used to arrange the sequence of image block context semantic feature vectors according to the corresponding spatial positions during the image segmentation processing to obtain the second-scale gait energy feature map.
[0102] More specifically, in the embodiment of the present application, the embedded coding unit 152 is used to: arrange the pixel values of all pixel positions in each image block in the sequence of the image blocks into a one-dimensional vector; and use the fully connected layer of the ViT model to fully connect the one-dimensional vector using the following fully connected coding formula to extract the high-dimensional implicit features of the feature values at each position in the one-dimensional vector, wherein the fully connected coding formula is: Where X is the one-dimensional vector, Y is the output vector, W is the weight matrix, and B is the bias vector. Represents matrix multiplication.
[0103] More specifically, in the embodiment of the present application, the multi-scale image feature association module 160 is used to fuse the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map, so as to fuse the multi-scale correlation feature distribution information based on long-distance dependency association and short-distance dependency association of the implicit features such as the gait speed and morphology of the patient during the training process in the bent leg training monitoring video.
[0104] In particular, in the technical solution of the present application, when the first-scale gait energy feature map and the second-scale gait energy feature map are fused to obtain the classification feature map, since the first-scale gait energy feature map and the second-scale gait energy feature map respectively express the image semantics of the gait energy map at different scales, that is, the first-scale gait energy feature map expresses the local correlation image semantics at the convolution kernel scale of the feature extractor, and the second-scale gait energy feature map expresses the contextual image semantics at the image block scale of the ViT model, it is expected to fuse the first-scale gait energy feature map and the second-scale gait energy feature map at different scales to enhance the fusion effect of the classification feature map on the first-scale gait energy feature map and the second-scale gait energy feature map. Based on this, the applicant of the present application performs Gaussian joint density fusion based on scale cognition on the first-scale gait energy feature map F1 and the second-scale gait energy feature map F2.
[0105] More specifically, in the embodiment of the present application, the multi-scale image feature association module 160 is used to: perform scale-aware Gaussian joint density fusion on the first-scale gait energy feature map and the second-scale gait energy feature map according to the following optimization formula to obtain the classification feature map; wherein the optimization formula is:
[0106]
[0107]
[0108] f1∈F1 and f2∈F2
[0109] Among them, F1 represents the first-scale gait energy feature map, F2 represents the second-scale gait energy feature map, f1 is the feature value of each position in the first-scale gait energy feature map, f2 is the feature value of each position in the second-scale gait energy feature map, μ and σ are the mean and variance of the feature set of all feature values of the first-scale gait energy feature map and the second-scale gait energy feature map, respectively, W, H and C are the width, height and number of channels of the feature map, respectively, and f′ is the feature value of each position of the classification feature map.
[0110] Here, the scale-aware Gaussian joint density fusion takes into account the scale-optimal expression characteristics of the features to be fused. In order to improve the effectiveness of feature fusion and the generalization performance relative to the features to be fused, the scale-aware Gaussian joint density is used as an advantage function to strategically represent the classification performance gap of feature distribution based on mean and variance, thereby improving the feature-scale self-dependence of feature fusion, so as to improve the fusion effect of the classification feature map on the first-scale gait energy feature map and the second-scale gait energy feature map. In this way, the gait detection and evaluation of patients during postoperative bent leg training can be accurately carried out, and abnormalities can be found in time to facilitate corresponding treatment.
[0111] More specifically, in the embodiment of the present application, the gait detection module 170 is used to pass the classification feature graph through a classifier to obtain a classification result, and the classification result is used to indicate whether the gait of the monitored object is normal. That is, in the technical solution of the present application, the label of the classifier includes the gait of the monitored object is normal (first label), and the gait of the monitored object is abnormal (second label), wherein the classifier determines which classification label the classification feature graph belongs to through a soft maximum function. It should be understood that in the technical solution of the present application, the classification label of the classifier is an evaluation detection label of whether the gait of the monitored object is normal. Therefore, after obtaining the classification result, the gait evaluation of the patient during the postoperative bent leg training can be accurately performed based on the classification result, and abnormalities can be found in time to facilitate corresponding processing.
[0112] It should be understood that the role of the classifier is to use the given categories and known training data to learn classification rules and classifiers, and then classify (or predict) unknown data. Logistic regression and SVM are often used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-classification. However, this is prone to errors and is not efficient. Commonly used multi-classification methods include the Softmax classification function.
[0113] More specifically, in the embodiments of the present application, Figure 4As shown, the gait detection module 170 includes: a feature map expansion unit 171, used to expand the classification feature map into a classification feature vector according to a row vector or a column vector; a fully connected encoding unit 172, used to use the fully connected layer of the classifier to fully connect the classification feature vector to obtain an encoded classification feature vector; and a classification unit 173, used to input the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.
[0114] In summary, the postoperative bent leg training monitoring system 100 based on the embodiment of the present application is explained, which first extracts multiple bent leg training monitoring key frames within a gait cycle from the bent leg training monitoring video of the monitored object, then processes the multiple bent leg training monitoring key frames to obtain a gait energy map, then passes the gait energy map through the convolutional neural network model and the ViT model respectively to obtain a first-scale gait energy feature map and a second-scale gait energy feature map, then fuses the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map, and finally passes the classification feature map through a classifier to obtain a classification result for indicating whether the gait of the monitored object is normal. In this way, the patient's gait assessment can be accurately performed, and abnormalities can be discovered in time to facilitate corresponding processing.
[0115] As described above, the postoperative bent leg training monitoring system 100 based on the embodiment of the present application can be implemented in various terminal devices, such as a server having a postoperative bent leg training monitoring algorithm based on the embodiment of the present application. In one example, the postoperative bent leg training monitoring system 100 based on the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the postoperative bent leg training monitoring system 100 based on the embodiment of the present application can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the postoperative bent leg training monitoring system 100 based on the embodiment of the present application can also be one of the many hardware modules of the terminal device.
[0116] Alternatively, in another example, the postoperative bent leg training monitoring system 100 based on the embodiment of the present application and the terminal device may also be separate devices, and the postoperative bent leg training monitoring system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.
[0117] Figure 5 FIG. 1 is a flow chart of a method for monitoring postoperative leg bending training according to an embodiment of the present application. Figure 5As shown, according to the postoperative bent leg training monitoring method of the embodiment of the present application, it includes: S110, obtaining a bent leg training monitoring video of the object to be monitored; S120, extracting multiple bent leg training monitoring key frames within a gait cycle from the bent leg training monitoring video of the object to be monitored; S130, processing the multiple bent leg training monitoring key frames to obtain a gait energy map; S140, passing the gait energy map through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature map; S150, passing the gait energy map through a ViT model to obtain a second-scale gait energy feature map; S160, fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and, S170, passing the classification feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether the gait of the object to be monitored is normal.
[0118] Figure 6 Schematic diagram of the system architecture of the postoperative leg bending training monitoring method according to the embodiment of the present application. Figure 6 As shown, in the system architecture of the postoperative bent leg training monitoring method, first, a bent leg training monitoring video of the object to be monitored is obtained; then, a plurality of bent leg training monitoring key frames within a gait cycle are extracted from the bent leg training monitoring video of the object to be monitored; then, the plurality of bent leg training monitoring key frames are processed to obtain a gait energy map; then, the gait energy map is passed through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature map; then, the gait energy map is passed through a ViT model to obtain a second-scale gait energy feature map; then, the first-scale gait energy feature map and the second-scale gait energy feature map are fused to obtain a classification feature map; finally, the classification feature map is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the gait of the object to be monitored is normal.
[0119] In a specific example, in the above-mentioned postoperative bent leg training monitoring method, the gait energy map is passed through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature map, including: each layer of the convolutional neural network model as a feature extractor performs the following in the forward pass of the layer: convolution processing on the input data to obtain a first-scale convolution feature map; mean pooling processing on the first-scale convolution feature map to obtain a first-scale pooling feature map; and, nonlinear activation on the first-scale pooling feature map to obtain a first-scale activation feature map; wherein the output of the last layer of the convolutional neural network model as a feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model as a feature extractor is the gait energy map.
[0120] In a specific example, in the above-mentioned postoperative bent leg training monitoring method, the gait energy map is passed through the ViT model to obtain a second-scale gait energy feature map, including: performing image block processing on the gait energy map to obtain a sequence of image blocks; using the embedding layer of the ViT model to embed and encode each image block in the sequence of image blocks to obtain a sequence of image block embedding vectors; inputting the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and arranging the sequence of image block context semantic feature vectors according to the corresponding spatial positions during the image block processing to obtain the second-scale gait energy feature map.
[0121] In a specific example, in the above-mentioned postoperative bent leg training monitoring method, the embedding layer of the ViT model is used to embed and encode each image block in the sequence of image blocks to obtain a sequence of image block embedding vectors, including: arranging the pixel values of all pixel positions in each image block in the sequence of image blocks into a one-dimensional vector; and using the fully connected layer of the ViT model to fully connect the one-dimensional vector using the following fully connected encoding formula to extract the high-dimensional implicit features of the eigenvalues at each position in the one-dimensional vector, wherein the fully connected encoding formula is: Where X is the one-dimensional vector, Y is the output vector, W is the weight matrix, and B is the bias vector. Represents matrix multiplication.
[0122] In a specific example, in the above-mentioned postoperative bent leg training monitoring method, fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map includes: performing scale-aware Gaussian joint density fusion on the first-scale gait energy feature map and the second-scale gait energy feature map using the following optimization formula to obtain the classification feature map; wherein the optimization formula is:
[0123]
[0124]
[0125] f1∈F1 and f2∈F2
[0126] Among them, F1 represents the first-scale gait energy feature map, F2 represents the second-scale gait energy feature map, f1 is the feature value of each position in the first-scale gait energy feature map, f2 is the feature value of each position in the second-scale gait energy feature map, μ and σ are the mean and variance of the feature set of all feature values of the first-scale gait energy feature map and the second-scale gait energy feature map, respectively, W, H and C are the width, height and number of channels of the feature map, respectively, and f′ is the feature value of each position of the classification feature map.
[0127] In a specific example, in the above-mentioned postoperative bent leg training monitoring method, the classification feature map is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the gait of the monitored object is normal, including: expanding the classification feature map into a classification feature vector according to a row vector or a column vector; using the fully connected layer of the classifier to fully connect encode the classification feature vector to obtain an encoded classification feature vector; and inputting the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.
[0128] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned postoperative bent leg training monitoring method have been described in detail above. Figures 1 to 4 It has been described in detail in the description of the postoperative bent leg training monitoring system 100, and therefore, its repeated description will be omitted.
[0129] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a computer, the above-mentioned method can be executed.
[0130] The program part of the technology can be considered as a "product" or "manufactured product" in the form of executable code and / or related data, which is participated in or realized by computer-readable media. Tangible and permanent storage media can include any memory or storage used by any computer, processor, or similar device or related module. For example, various semiconductor memories, tape drives, disk drives or any similar devices that can provide storage functions for software.
[0131] All or part of the software may sometimes communicate over a network, such as the Internet or other communications network. Such communications can load software from one computer device or processor to another. For example: from a server or host computer of a video target detection device to a hardware platform of a computer environment, or other computer environment that implements the system, or a system with similar functions related to providing information required for target detection. Therefore, another medium capable of transmitting software elements can also be used as a physical connection between local devices, such as light waves, radio waves, electromagnetic waves, etc., transmitted through cables, optical cables or air. Physical media used to carry carriers, such as cables, wireless connections or optical cables and the like, can also be considered as media that carry software. As used herein, unless limited to tangible "storage" media, other terms referring to computer or machine "readable media" refer to media involved in the process of the processor executing any instructions.
[0132] The present application uses specific words to describe the embodiments of the present application. For example, "first / second embodiment", "one embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or multiple times in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.
[0133] In addition, it will be appreciated by those skilled in the art that various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of the present application may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.
[0134] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.
[0135] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.
Claims
1. A postoperative bent leg training monitoring system, characterized in that: include: A video data acquisition module, used for acquiring a monitoring video of a bent leg training of a subject to be monitored; A key frame extraction module, used for extracting a plurality of bent-leg training monitoring key frames within a gait cycle from the bent-leg training monitoring video of the subject to be monitored; A gait energy graph generating module, used for processing the plurality of bent-leg training monitoring key frames to obtain a gait energy graph; A first scale image feature extraction module, used for passing the gait energy graph through a convolutional neural network model as a feature extractor to obtain a first scale gait energy feature graph; A second-scale image feature extraction module, used for passing the gait energy map through the ViT model to obtain a second-scale gait energy feature map; A multi-scale image feature association module, used for fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; as well as A gait detection module, used for passing the classification feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the gait of the monitored object is normal; The multi-scale image feature association module is used to: Performing scale-aware Gaussian joint density fusion on the first-scale gait energy feature map and the second-scale gait energy feature map using the following optimization formula to obtain the classification feature map; Wherein, the optimization formula is: in, represents the first scale gait energy feature map, represents the second scale gait energy feature map, is the eigenvalue of each position in the first scale gait energy feature map, is the eigenvalue of each position in the second scale gait energy feature map, and are the mean and variance of the feature set of all eigenvalues of the first-scale gait energy feature map and the second-scale gait energy feature map, respectively, , and are the width, height and number of channels of the feature map, respectively, and are the feature values of each position of the classification feature map; The first-scale image feature extraction module is used to: Each layer of the convolutional neural network model as a feature extractor is performed separately in the forward pass of the layer: Performing convolution processing on the input data to obtain a first-scale convolution feature map; Performing mean pooling processing on the first-scale convolutional feature map to obtain a first-scale pooled feature map; and Performing nonlinear activation on the first-scale pooling feature map to obtain a first-scale activation feature map; The output of the last layer of the convolutional neural network model as a feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model as a feature extractor is the gait energy map; The second scale image feature extraction module comprises: An image block unit, used for performing image block processing on the gait energy map to obtain a sequence of image blocks; An embedding coding unit, configured to use the embedding layer of the ViT model to perform embedding coding on each image block in the sequence of image blocks to obtain a sequence of image block embedding vectors; An image context semantic association unit, configured to input the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and A dimensionality reconstruction unit is used to arrange the sequence of the image block context semantic feature vectors according to the corresponding spatial positions during the image block processing to obtain the second-scale gait energy feature map.
2. The postoperative bent leg training monitoring system according to claim 1, characterized in that: The embedded coding unit is used for: Arranging pixel values of all pixel positions in each image block in the sequence of image blocks into a one-dimensional vector; and The one-dimensional vector is fully connected encoded using the fully connected layer of the ViT model using the following fully connected encoding formula to extract high-dimensional implicit features of the eigenvalues at each position in the one-dimensional vector, wherein the fully connected encoding formula is: ,in, is the one-dimensional vector, is the output vector, is the weight matrix, is the bias vector, Represents matrix multiplication.
3. The postoperative bent leg training monitoring system according to claim 2, characterized in that: The gait detection module comprises: A feature map expansion unit, used to expand the classification feature map into a classification feature vector according to a row vector or a column vector; a fully connected encoding unit, configured to perform fully connected encoding on the classification feature vector using the fully connected layer of the classifier to obtain an encoded classification feature vector; and The classification unit is used to input the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.
4. A method for monitoring postoperative bent leg training, characterized in that: include: Obtaining a monitoring video of a bent-leg training of a subject to be monitored; Extracting a plurality of bent-leg training monitoring key frames within a gait cycle from the bent-leg training monitoring video of the subject to be monitored; Processing the plurality of bent-leg training monitoring key frames to obtain a gait energy graph; Passing the gait energy graph through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature graph; Passing the gait energy graph through the ViT model to obtain a second-scale gait energy feature graph; fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; as well as Passing the classification feature graph through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the gait of the monitored object is normal; Performing scale-aware Gaussian joint density fusion on the first-scale gait energy feature map and the second-scale gait energy feature map using the following optimization formula to obtain the classification feature map; Wherein, the optimization formula is: in, represents the first scale gait energy feature map, represents the second scale gait energy feature map, is the eigenvalue of each position in the first scale gait energy feature map, is the eigenvalue of each position in the second scale gait energy feature map, and are the mean and variance of the feature set of all eigenvalues of the first-scale gait energy feature map and the second-scale gait energy feature map, respectively, , and are the width, height and number of channels of the feature map, respectively, and are the feature values of each position of the classification feature map; The gait energy graph is passed through a convolutional neural network model as a feature extractor to obtain a first-scale gait energy feature graph, including: Each layer of the convolutional neural network model as a feature extractor is performed separately in the forward pass of the layer: Performing convolution processing on the input data to obtain a first-scale convolution feature map; Performing mean pooling processing on the first-scale convolutional feature map to obtain a first-scale pooled feature map; and Performing nonlinear activation on the first-scale pooling feature map to obtain a first-scale activation feature map; The output of the last layer of the convolutional neural network model as a feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model as a feature extractor is the gait energy map; The gait energy graph is passed through the ViT model to obtain a second-scale gait energy feature graph, including: Performing image block processing on the gait energy map to obtain a sequence of image blocks; Using the embedding layer of the ViT model, embedding coding is performed on each image block in the sequence of image blocks to obtain a sequence of image block embedding vectors; Inputting the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block contextual semantic feature vectors; and The sequence of the image block context semantic feature vectors is arranged according to the corresponding spatial positions during the image block processing to obtain the second-scale gait energy feature map.
5. The postoperative bent leg training monitoring method according to claim 4, characterized in that: Using the embedding layer of the ViT model to embed the image blocks in the sequence of image blocks to obtain a sequence of image block embedding vectors, including: Arranging pixel values of all pixel positions in each image block in the sequence of image blocks into a one-dimensional vector; and The one-dimensional vector is fully connected encoded using the fully connected layer of the ViT model using the following fully connected encoding formula to extract high-dimensional implicit features of the eigenvalues at each position in the one-dimensional vector, wherein the fully connected encoding formula is: ,in, is the one-dimensional vector, is the output vector, is the weight matrix, is the bias vector, Represents matrix multiplication.
