Leg hyperthermia rehabilitation equipment

Abstract

Disclosed is a leg thermotherapy rehabilitation device. Using a first convolutional neural network model with a temporal attention mechanism, the device mines hidden features of images from a patient's leg thermotherapy monitoring video over a predetermined time period to extract characteristic information about the temporal changes in the patient's leg skin color. A multi-scale neighborhood feature extraction module extracts multi-scale neighborhood correlation features of the device's heater power values ​​at different time spans within the predetermined time period. Based on the state change characteristics of the patient's leg thermotherapy, the device adaptively adjusts the heater power, dynamically controlling the thermotherapy temperature in real time based on actual conditions. This intelligently controls the thermotherapy temperature to prevent burns and secondary injuries to the patient.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology, and more specifically, to a leg thermal therapy rehabilitation device. Background Art

[0002] Bone tumors are tumors that develop in bones or their associated tissues. They can be benign or malignant. Benign bone tumors are easily curable and have a good prognosis, while malignant bone tumors progress rapidly, have a poor prognosis, and a high mortality rate. Among them, bone tumors are most commonly found in the leg joints. Currently, the treatment for leg joint bone tumors involves surgical resection of the leg tumor. After the tumor is removed, the patient's leg needs to undergo rehabilitation treatment.

[0003] Existing leg hyperthermia rehabilitation programs heat the legs, which can lead to overheating and uncontrolled burns. Therefore, there is a need for optimized leg hyperthermia rehabilitation equipment that can adjust the temperature based on the actual location of the patient's leg, ensuring effective therapy while preventing burns. Summary of the Invention

[0004] In order to solve the above technical problems, the present application is proposed. An embodiment of the present application provides a leg thermotherapy rehabilitation device, which mines hidden features of images on a patient's leg thermotherapy monitoring video in a predetermined time period by using a first convolutional neural network model with a time attention mechanism, so as to extract the state change feature information of the patient's leg skin color in the time dimension; extracts the multi-scale neighborhood correlation features of the power value of the heater of the leg thermotherapy rehabilitation device under different time spans within a predetermined time period through a multi-scale neighborhood feature extraction module; then, based on the state change features of the patient's leg thermotherapy, the heating power of the heater of the thermotherapy rehabilitation device is adaptively adjusted, and thereby the thermotherapy temperature is dynamically controlled in real time based on actual conditions. In this way, the temperature of the thermotherapy can be intelligently controlled to avoid secondary injuries caused by burns to the patient.

[0005] According to one aspect of the present application, there is provided a leg thermal therapy rehabilitation device, comprising:

[0006] a data monitoring unit, configured to obtain power values ​​of a heater of a leg hyperthermia rehabilitation device at a plurality of predetermined time points within a predetermined time period and a monitoring video of the leg hyperthermia during the predetermined time period captured by a camera;

[0007] a sampling unit, configured to extract a plurality of hyperthermia monitoring key frames from the leg hyperthermia monitoring video;

[0008] a reflected light component extraction unit, configured to extract the reflected light component of each pixel of each of the plurality of hyperthermia monitoring key frames to obtain a plurality of hyperthermia monitoring reflected light component key frames;

[0009] A temporal attention encoding unit is configured to obtain a thermotherapy monitoring feature map by encoding the plurality of thermotherapy monitoring reflected light component key frames through a first convolutional neural network model using a temporal attention mechanism;

[0010] a dimension adjustment unit, configured to perform global mean pooling on each feature matrix along the channel dimension of the hyperthermia monitoring feature map to obtain a hyperthermia monitoring feature vector;

[0011] A power feature extraction unit, configured to arrange the power values ​​of the plurality of predetermined time points into a power input vector according to the time dimension and then obtain a multi-scale power feature vector by a multi-scale neighborhood feature extraction module;

[0012] a responsiveness estimation unit, configured to calculate a responsiveness estimation of the hyperthermia monitoring feature vector relative to the multi-scale power feature vector to obtain a classification feature matrix; and

[0013] The control result generating unit is used to pass the classification feature matrix through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the power value of the heater at the current time point should be increased or decreased.

[0014] In the above-mentioned leg hyperthermia rehabilitation device, the sampling unit is further used to extract the multiple hyperthermia monitoring key frames from the leg hyperthermia monitoring video at a predetermined sampling frequency.

[0015] In the above-mentioned leg hyperthermia rehabilitation device, the reflected light component extraction unit is further used to extract the reflected light component of each pixel of each hyperthermia monitoring key frame in the multiple hyperthermia monitoring key frames using the following formula to obtain the multiple hyperthermia monitoring reflected light component key frames; wherein the formula is:

[0016] S(x, y) = R(x, y) L(x, y)

[0017] Wherein, S(x, y) represents each pixel value of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames, R(x, y) represents each pixel of each thermotherapy monitoring reflected light component key frame in the multiple thermotherapy monitoring reflected light component key frames, L(x, y) represents each pixel of each thermotherapy monitoring brightness component key frame in the multiple thermotherapy monitoring brightness component key frames, and the symbol · represents matrix multiplication.

[0018] In the above-mentioned leg thermotherapy rehabilitation device, the temporal attention encoding unit includes: an adjacent frame extraction unit, used to extract the adjacent first frame and second frame from the multiple thermotherapy monitoring reflected light component key frames; a first image local feature extraction unit, used to pass the first frame and the second frame respectively through the first convolution layer and the second convolution layer of the first convolutional neural network model using the temporal attention mechanism to obtain a first frame feature map corresponding to the first frame and a second frame feature map corresponding to the second frame; a temporal attention unit, used to perform positional point multiplication on the first frame feature map and the second frame feature map and then pass the Softmax activation function to obtain a temporal attention map; a second image local feature extraction unit, used to pass the second frame through the third convolution layer of the first convolutional neural network model using the temporal attention mechanism to obtain a second frame initial feature map; and a temporal attention applying unit, used to perform positional point multiplication on the second frame initial feature map and the temporal attention map to obtain a thermotherapy monitoring feature map.

[0019] In the above-mentioned leg thermal therapy rehabilitation device, the power feature extraction unit includes: a first scale feature extraction unit, used to input the power input vector into the first convolution layer of the multi-scale neighborhood feature extraction module to obtain a first scale power feature vector, wherein the first convolution layer has a first one-dimensional convolution kernel of a first length; a second scale feature extraction unit, used to input the power input vector into the second convolution layer of the multi-scale neighborhood feature extraction module to obtain a second scale power feature vector, wherein the second convolution layer has a second one-dimensional convolution kernel of a second length, and the first length is different from the second length; and a multi-scale cascade unit, used to cascade the first scale power feature vector and the second scale power feature vector to obtain the multi-scale power feature vector.

[0020] In the above-mentioned leg hyperthermia rehabilitation device, the first-scale feature extraction unit is further used to: use the first convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the power input vector using the following formula to obtain the first-scale power feature vector; wherein the formula is:

[0021]

[0022] Wherein, a is the width of the first convolution kernel in the x direction, F(a) is the first convolution kernel parameter vector, G(xa) is the local vector matrix operated with the convolution kernel function, w is the size of the first convolution kernel, and X represents the power input vector.

[0023] The second-scale feature extraction unit is further configured to: use the second convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the power input vector using the following formula to obtain the second-scale power feature vector; wherein the formula is:

[0024]

[0025] Wherein, b is the width of the second convolution kernel in the x direction, F(b) is the second convolution kernel parameter vector, G(xb) is the local vector matrix operated with the convolution kernel function, m is the size of the second convolution kernel, and X represents the power input vector.

[0026] In the above-mentioned leg thermal therapy rehabilitation device, the responsiveness estimation unit includes: a Gaussian density map construction subunit, used to construct a self-Gaussian density map of the thermal therapy monitoring feature vector and the multi-scale power feature vector to obtain a first self-Gaussian density map and a second self-Gaussian density map; a responsiveness subunit, used to calculate the responsiveness estimation of the first self-Gaussian density map relative to the second self-Gaussian density map to obtain a responsiveness Gaussian density map; and a Gaussian discretization subunit, used to perform Gaussian discretization on the Gaussian distribution of each position of the responsiveness Gaussian density map to obtain the classification feature matrix.

[0027] In the above-mentioned leg thermal therapy rehabilitation device, the control result generating unit is further used to: use the classifier to process the classification feature matrix according to the following formula to generate a classification result; wherein, the formula is: softmax{(M c , B c )|Project(F)}, where Project(F) represents projecting the classification feature matrix into a vector, M c is the weight matrix of the fully connected layer, B c Represents the bias vector of the fully connected layer.

[0028] The above-mentioned leg thermotherapy rehabilitation device also includes a training module for training the first convolutional neural network model using the time attention mechanism, the multi-scale neighborhood feature extraction module and the classifier; wherein the training module includes: a training data acquisition unit for acquiring training data, the training data including the training power value of the leg thermotherapy rehabilitation device at multiple predetermined time points within a predetermined time period, the leg thermotherapy training monitoring video of the predetermined time period, and the actual value of the power value of the heater at the current time point that should be increased or decreased; a training sampling unit for extracting multiple training thermotherapy monitoring key frames from the leg thermotherapy training monitoring video; a training reflected light component extraction unit for extracting the reflected light component of each pixel of each training thermotherapy monitoring key frame in the multiple training thermotherapy monitoring key frames to obtain multiple training thermotherapy monitoring reflected light component key frames; a training time attention encoding unit for passing the multiple training thermotherapy monitoring reflected light component key frames through the first convolutional neural network model using the time attention mechanism to obtain a training thermotherapy monitoring feature map; a training dimension adjustment unit for The present invention relates to a method for preparing a first convolutional neural network model using a temporal attention mechanism, wherein the first convolutional neural network model comprises the following steps: performing global mean pooling on each feature matrix along the channel dimension of the thermotherapy monitoring feature map to obtain a training thermotherapy monitoring feature vector; a training power feature extraction unit for arranging the training power values ​​of the multiple predetermined time points into a training power input vector according to the time dimension and then passing it through the multi-scale neighborhood feature extraction module to obtain a training multi-scale power feature vector; a training responsiveness estimation unit for calculating the responsiveness estimation of the training thermotherapy monitoring feature vector relative to the training multi-scale power feature vector to obtain a training classification feature matrix; a classification loss unit for passing the training classification feature matrix through the classifier to obtain a classification loss function value; an internalized learning loss unit for calculating the sequence-to-sequence response rule internalized learning loss function value based on the distance between the training thermotherapy monitoring feature vector and the training multi-scale power feature vector; and a training unit for calculating the weighted sum of the classification loss function value and the sequence-to-sequence response rule internalized learning loss function value as a loss function value to train the first convolutional neural network model using the temporal attention mechanism, the multi-scale neighborhood feature extraction module and the classifier.

[0029] In the above-mentioned leg hyperthermia rehabilitation device, the internalized learning loss unit is further used to calculate the sequence-to-sequence response rule internalized learning loss function value based on the distance between the training hyperthermia monitoring feature vector and the training multi-scale power feature vector using the following formula; wherein the formula is:

[0030]

[0031]

[0032]

[0033] Wherein, V1 is the training hyperthermia monitoring feature vector, V2 is the training multi-scale power feature vector, and W1 and W2 are the weight matrices of the classifier for the training hyperthermia monitoring feature vector and the training multi-scale power feature vector, respectively. ReLU(·) represents the ReLU activation function, and Sigmoid(·) represents the Sigmoid activation function. represents matrix multiplication, and d(·,·) represents the Euclidean distance between two vectors.

[0034] Compared with the existing technology, the leg thermotherapy rehabilitation device provided by this application uses a first convolutional neural network model with a temporal attention mechanism to mine the hidden features of the image in a predetermined time period of the patient's leg thermotherapy monitoring video to extract the state change characteristic information of the patient's leg skin color in the time dimension; a multi-scale neighborhood feature extraction module is used to extract the multi-scale neighborhood correlation features of the power value of the heater of the leg thermotherapy rehabilitation device at different time spans within the predetermined time period; then, based on the state change characteristics of the patient's leg thermotherapy, the heating power of the heater of the thermotherapy rehabilitation device is adaptively adjusted, and the thermotherapy temperature is dynamically controlled in real time based on the actual situation. In this way, the thermotherapy temperature can be intelligently controlled to avoid burns on the patient and secondary injuries. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0036] Figure 1 This is an application scenario diagram of the leg thermal therapy rehabilitation equipment according to an embodiment of the present application.

[0037] Figure 2 This is a block diagram of a leg thermal therapy rehabilitation device according to an embodiment of the present application.

[0038] Figure 3 This is a block diagram of the time attention encoding unit in the leg thermal therapy rehabilitation device according to an embodiment of the present application.

[0039] Figure 4 4 is a block diagram of the power feature extraction unit in the leg thermal therapy rehabilitation device according to an embodiment of the present application.

[0040] Figure 54 is a block diagram of the responsiveness estimation unit in the leg thermal therapy rehabilitation device according to an embodiment of the present application.

[0041] Figure 6 This is a block diagram of the training module in the leg thermal therapy rehabilitation device according to an embodiment of the present application.

[0042] Figure 7 This is a flow chart of a method for using the leg thermal therapy rehabilitation device according to an embodiment of the present application.

[0043] Figure 8 Schematic diagram of the system architecture of the method for using the leg thermal therapy rehabilitation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0045] Scenario Overview

[0046] As mentioned above, bone tumors are tumors that develop in the bones or their appendages. They can be benign or malignant. Benign bone tumors are easily curable and have a good prognosis, while malignant bone tumors progress rapidly, have a poor prognosis, and a high mortality rate. Among them, bone tumors are most commonly found in the leg joints. Currently, the treatment for leg joint bone tumors involves surgical resection of the leg tumor. After the tumor is removed, the patient's leg needs to undergo rehabilitation treatment.

[0047] Existing leg hyperthermia rehabilitation programs heat the legs, which can lead to overheating and uncontrolled burns. Therefore, there is a need for optimized leg hyperthermia rehabilitation equipment that can adjust the temperature based on the actual location of the patient's leg, ensuring effective therapy while preventing burns.

[0048] Accordingly, considering that if the temperature is too high, burns may occur, while if the temperature is too low, the treatment effect may be insufficient, the key to optimizing leg thermal therapy rehabilitation equipment lies in the temperature control strategy. The temperature control of the leg thermal therapy rehabilitation equipment should be adapted to the changing state of the patient's leg thermal therapy. In other words, the heating power of the thermal therapy rehabilitation equipment should be adaptively adjusted based on the changing characteristics of the patient's leg thermal therapy state. In this way, the thermal therapy temperature can be dynamically controlled in real time based on the actual situation to avoid burns and secondary injuries to the patient.

[0049] Specifically, in the technical solution of the present application, first, the power value of the heater of the leg thermotherapy rehabilitation device at multiple predetermined time points within a predetermined time period is obtained, and the leg thermotherapy monitoring video of the predetermined time period is collected by a camera. Next, considering that in the leg thermotherapy monitoring video of the predetermined time period, the state change characteristics of the patient's legs can be represented by the difference between adjacent monitoring frames in the monitoring video, that is, the change in the state of the patient's legs is represented by the image representation of adjacent image frames. However, considering that the difference between adjacent frames in the monitoring video is small and there is a large amount of data redundancy, in order to reduce the measurement and avoid the adverse effects of data redundancy on detection, the leg thermotherapy monitoring video is key frame sampled at a predetermined sampling frequency to extract the multiple thermotherapy monitoring key frames. Here, it is worth mentioning that the sampling frequency can be adjusted accordingly based on the power value of the heater of the leg thermotherapy rehabilitation device in the actual scenario, rather than the default value.

[0050] It should be understood that when performing thermotherapy, more attention is paid to the color changes of the patient's legs, because temperature will cause the skin of the legs to change color, and considering that the color of an object is determined by the object's ability to reflect light of different wavelengths. Therefore, in the technical solution of the present application, the reflected light component of each pixel of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames is further extracted to obtain multiple thermotherapy monitoring reflected light component key frames. In this way, the problems of small image dynamic range and small improvement in color richness can be solved, and the skin color details in the thermotherapy monitoring frame can also be restored to a considerable extent, which is conducive to the subsequent improvement of the skin color change characteristics of the patient's legs, that is, the accuracy and richness of the extraction of the thermotherapy state change characteristics of the patient's legs.

[0051] Then, for the multiple thermotherapy monitoring reflected light component keyframes in the surveillance video, a convolutional neural network model, which excels in extracting latent features from images, is used to mine hidden features of the image. Specifically, in order to more accurately and fully extract the characteristic distribution information of the skin color changes on the patient's legs, a first convolutional neural network model with a temporal attention mechanism is further used to extract features from each of the multiple thermotherapy monitoring reflected light component keyframes. This extracts characteristic information about the dynamic changes in the skin color of the patient's legs over time, thereby generating a thermotherapy monitoring feature map.

[0052] Next, global mean pooling processing is performed on each feature matrix along the channel dimension of the thermotherapy monitoring feature map to retain the characteristic information of the temporal change of the skin color of the patient's legs in the thermotherapy monitoring feature map when performing data feature dimensionality reduction, so as to reduce the amount of calculation and prevent overfitting, thereby obtaining a thermotherapy monitoring feature vector.

[0053] Furthermore, considering that the power value of the heater of the leg thermal therapy rehabilitation device has different pattern characteristics under different time period spans, in order to accurately extract the implicit change characteristic information of the power value of the heater, the power values ​​of the multiple predetermined time points are further arranged as power input vectors according to the time dimension and then encoded in a multi-scale neighborhood feature extraction module to extract the multi-scale neighborhood correlation features of the power values ​​under different time spans within the predetermined time period, thereby obtaining a multi-scale power feature vector.

[0054] Then, the responsiveness estimate of the thermotherapy monitoring feature vector relative to the multi-scale power feature vector is calculated to represent the correlation characteristics between the state change characteristics of the thermotherapy of the patient's leg and the dynamic change characteristics of the heating power of the thermotherapy device, and classification is performed based on this to obtain the classification result of the heating power value adjustment at the current time point. In particular, considering the volatility and uncertainty of the heating power value, and the volatility and uncertainty of the state change of the thermotherapy of the patient's leg as the heating power changes, in order to improve the accuracy of the power control of the heater at the current time point based on the actual situation of the patient's leg, it is necessary to perform data enhancement on the state change characteristics of the thermotherapy of the patient's leg and the dynamic change characteristics of the heating power of the thermotherapy device in a high-dimensional feature space.

[0055] It should be understood that as a learning target for a neural network model, a Gaussian density map can represent the joint distribution of a single eigenvalue of a feature distribution due to its probability density when multiple eigenvalues ​​constitute an overall distribution. That is, using the feature distribution as a prior distribution, the probability density at each prior distribution position due to the correlation of other prior distribution positions is obtained as a posterior distribution, thereby more accurately describing the feature distribution in a higher dimension. Therefore, in the technical solution of the present application, a self-Gaussian density map of the hyperthermia monitoring feature vector and the multi-scale power feature vector is constructed for data enhancement, thereby obtaining a first self-Gaussian density map and a second self-Gaussian density map.

[0056] Furthermore, the responsiveness estimate of the first self-Gaussian density map relative to the second self-Gaussian density map can be calculated to represent the correlation characteristics between the state change characteristics of the patient's leg thermal therapy and the dynamic change characteristics of the heating power of the thermal therapy device, thereby obtaining a responsive Gaussian density map. Next, the Gaussian distribution at each position of the responsive Gaussian density map is Gaussian discretized to avoid information loss during data feature augmentation, thereby obtaining the classification feature matrix. The classification feature matrix is ​​then used to perform classification processing in a classifier to obtain a classification result indicating whether the power value of the heater at the current time point should be increased or decreased.

[0057] In particular, in the technical solution of the present application, by calculating the responsiveness estimate of the thermal therapy monitoring feature vector relative to the multi-scale power feature vector based on the Gaussian density map, the obtained classification feature matrix can reflect the responsiveness relationship between the overall feature probability density distribution of the thermal therapy monitoring feature vector and the multi-scale power feature vector.

[0058] Preferably, to further enhance the accuracy of the responsiveness calculation between the hyperthermia monitoring feature vector and the multi-scale power feature vector, the eigenvalue sequence distribution characteristics of the hyperthermia monitoring feature vector and the multi-scale power feature vector can be used to learn the cause-effect response between the sequences under a predetermined classification task. Specifically, a sequence-to-sequence response rule internalization learning loss function is introduced, expressed as:

[0059]

[0060]

[0061]

[0062] V1 and V2 are the hyperthermia monitoring feature vector and the multi-scale power feature vector, respectively, and M1 and M2 are the weight matrices of the classifier for V1 and V2, respectively.

[0063] Here, the classifier uses a squeeze-and-excite channel attention mechanism for the weight matrices of different sequences to obtain enhanced discriminative capabilities between sequences. By training the network with this loss function, it is possible to restore more discriminative causal features between response sequences, thereby internalizing the cause-and-effect response rules between sequences. This enhances the accuracy of responsiveness calculations between sequences and improves classification accuracy. In this way, the heating power of the thermal therapy rehabilitation device can be adaptively adjusted based on the changes in the skin condition of the patient's leg during thermal therapy, thereby controlling the temperature of the thermal therapy and preventing the patient from being burned and causing secondary injuries.

[0064] Based on this, the present application provides a leg thermotherapy rehabilitation device, which includes: a data monitoring unit, used to obtain the power value of the heater of the leg thermotherapy rehabilitation device at multiple predetermined time points within a predetermined time period and the leg thermotherapy monitoring video of the predetermined time period collected by a camera; a sampling unit, used to extract multiple thermotherapy monitoring key frames from the leg thermotherapy monitoring video; a reflected light component extraction unit, used to extract the reflected light component of each pixel of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames to obtain multiple thermotherapy monitoring reflected light component key frames; a time attention encoding unit, used to encode the multiple thermotherapy monitoring reflected light component key frames by using a first convolutional neural network model of the time attention mechanism to obtain a thermotherapy monitoring key frame. control feature map; a dimension adjustment unit, used to perform global mean pooling on each feature matrix along the channel dimension of the thermotherapy monitoring feature map to obtain a thermotherapy monitoring feature vector; a power feature extraction unit, used to arrange the power values ​​of the multiple predetermined time points into a power input vector according to the time dimension and then pass it through a multi-scale neighborhood feature extraction module to obtain a multi-scale power feature vector; a responsiveness estimation unit, used to calculate the responsiveness estimate of the thermotherapy monitoring feature vector relative to the multi-scale power feature vector to obtain a classification feature matrix; and a control result generation unit, used to pass the classification feature matrix through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the power value of the heater at the current time point should be increased or decreased.

[0065] Figure 1 FIG is an application scenario diagram of the leg thermal therapy rehabilitation device according to an embodiment of the present application. Figure 1 As shown, in this application scenario, first, the leg thermal therapy rehabilitation device (for example, Figure 1 The power value of the heater (e.g., T) shown in FIG Figure 1 C1 shown in the figure), and through a camera (e.g. Figure 1 M) collects the leg thermal therapy monitoring video of the predetermined time period (for example, Figure 1 Then, the power value of the heater and the leg hyperthermia monitoring video are input into the server where the algorithm of the leg hyperthermia rehabilitation device is deployed (for example, Figure 1 S) as shown in , wherein the server is capable of processing the power value of the heater and the leg thermotherapy monitoring video using an algorithm of the leg thermotherapy rehabilitation device to generate a classification result indicating whether the power value of the heater at the current time point should be increased or decreased.

[0066] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0067] Exemplary Systems

[0068] Figure 2 FIG is a block diagram of a leg thermal therapy rehabilitation device according to an embodiment of the present application. Figure 2 As shown, the leg thermotherapy rehabilitation device 100 according to an embodiment of the present application includes: a data monitoring unit 110, which is used to obtain the power value of the heater of the leg thermotherapy rehabilitation device at multiple predetermined time points within a predetermined time period and the leg thermotherapy monitoring video of the predetermined time period collected by the camera; a sampling unit 120, which is used to extract multiple thermotherapy monitoring key frames from the leg thermotherapy monitoring video; a reflected light component extraction unit 130, which is used to extract the reflected light component of each pixel of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames to obtain multiple thermotherapy monitoring reflected light component key frames; a time attention encoding unit 140, which is used to encode the multiple thermotherapy monitoring reflected light component key frames by using a first convolutional neural network model of the time attention mechanism to obtain thermotherapy A monitoring feature map; a dimension adjustment unit 150, configured to perform global mean pooling on each feature matrix along the channel dimension of the thermotherapy monitoring feature map to obtain a thermotherapy monitoring feature vector; a power feature extraction unit 160, configured to arrange the power values ​​of the plurality of predetermined time points into a power input vector according to the time dimension and then pass the power through a multi-scale neighborhood feature extraction module to obtain a multi-scale power feature vector; a responsiveness estimation unit 170, configured to calculate a responsiveness estimate of the thermotherapy monitoring feature vector relative to the multi-scale power feature vector to obtain a classification feature matrix; and a control result generation unit 180, configured to pass the classification feature matrix through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the power value of the heater at the current time point should be increased or decreased.

[0069] Specifically, in an embodiment of the present application, the data monitoring unit 110 and the sampling unit 120 are used to obtain the power value of the heater of the leg thermotherapy rehabilitation equipment at multiple predetermined time points within a predetermined time period and the leg thermotherapy monitoring video of the predetermined time period captured by the camera; and to extract multiple thermotherapy monitoring key frames from the leg thermotherapy monitoring video. As mentioned above, bone tumors are tumors that occur in bones or their appendages, and are divided into benign and malignant. Benign bone tumors are easy to cure and have a good prognosis, while malignant bone tumors develop rapidly, have a poor prognosis, and a high mortality rate. Among them, the common site of bone tumors is the leg joints. At present, the treatment plan for bone tumors in the leg joints is to surgically remove the tumor in the leg. After the tumor is removed, the patient's leg needs to undergo recovery treatment.

[0070] Existing leg hyperthermia rehabilitation programs heat the legs, which can lead to overheating and uncontrolled burns. Therefore, there is a need for optimized leg hyperthermia rehabilitation equipment that can adjust the temperature based on the actual location of the patient's leg, ensuring effective therapy while preventing burns.

[0071] Accordingly, considering that if the temperature is too high, burns may occur, while if the temperature is too low, the treatment effect may be insufficient, the key to optimizing leg thermal therapy rehabilitation equipment lies in the temperature control strategy. The temperature control of the leg thermal therapy rehabilitation equipment should be adapted to the changing state of the patient's leg thermal therapy. In other words, the heating power of the thermal therapy rehabilitation equipment should be adaptively adjusted based on the changing characteristics of the patient's leg thermal therapy state. In this way, the thermal therapy temperature can be dynamically controlled in real time based on the actual situation to avoid burns and secondary injuries to the patient.

[0072] Specifically, in the technical solution of the present application, first, the power values ​​of the heater of the leg thermotherapy rehabilitation device are obtained at multiple predetermined time points within a predetermined time period, and a leg thermotherapy monitoring video for the predetermined time period is captured by a camera. Next, considering that the state change characteristics of the patient's legs in the leg thermotherapy monitoring video for the predetermined time period can be represented by the difference between adjacent monitoring frames in the monitoring video, that is, the change in the patient's leg state is represented by the image representation of adjacent image frames.

[0073] However, considering that the differences between adjacent frames in the surveillance video are small and there is a large amount of data redundancy, in order to reduce measurement and avoid the adverse effects of data redundancy on detection, key frame sampling is performed on the leg hyperthermia monitoring video at a predetermined sampling frequency to extract the multiple hyperthermia monitoring key frames. It is worth mentioning that the sampling frequency can be adjusted accordingly based on the power value of the heater of the leg hyperthermia rehabilitation device in actual scenarios, rather than the default value.

[0074] Specifically, in this embodiment of the present application, the reflected light component extraction unit 130 is configured to extract the reflected light component of each pixel in each of the multiple thermotherapy monitoring keyframes to obtain multiple thermotherapy monitoring reflected light component keyframes. It should be understood that during thermotherapy, the color change of the patient's legs is of particular interest, as temperature causes the skin color of the legs to change. Considering that the color of an object is determined by its ability to reflect light of different wavelengths, the technical solution of the present application further extracts the reflected light component of each pixel in each of the multiple thermotherapy monitoring keyframes to obtain multiple thermotherapy monitoring reflected light component keyframes. This solves the problems of limited image dynamic range and limited improvement in color richness, and also restores a considerable degree of skin color detail in the thermotherapy monitoring frames, facilitating subsequent improvement in the accuracy and richness of the extracted skin color change features of the patient's legs, i.e., the features of the thermotherapy state change of the patient's legs.

[0075] Furthermore, the reflected light component of each pixel of each of the multiple hyperthermia monitoring key frames is extracted using the following formula to obtain the multiple hyperthermia monitoring reflected light component key frames; wherein the formula is:

[0076] S(x, y) = R(x, y) L(x, y)

[0077] Wherein, S(x, y) represents each pixel value of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames, R(x, y) represents each pixel of each thermotherapy monitoring reflected light component key frame in the multiple thermotherapy monitoring reflected light component key frames, L(x, y) represents each pixel of each thermotherapy monitoring brightness component key frame in the multiple thermotherapy monitoring brightness component key frames, and the symbol · represents matrix multiplication.

[0078] Specifically, in an embodiment of the present application, the time attention encoding unit 140 is used to obtain a thermotherapy monitoring feature map by using the first convolutional neural network model of the time attention mechanism for the multiple thermotherapy monitoring reflected light component key frames. Then, for the multiple thermotherapy monitoring reflected light component key frames in the monitoring video, a convolutional neural network model with excellent performance in extracting implicit features of the image is used to mine the hidden features of the image. In particular, in order to be able to more accurately and fully extract the characteristic distribution information of the skin color change of the patient's legs, the first convolutional neural network model of the time attention mechanism is further used to perform feature extraction on each of the multiple thermotherapy monitoring reflected light component key frames, so as to extract the dynamic change characteristic information of the skin color of the patient's legs in the time dimension, thereby obtaining a thermotherapy monitoring feature map.

[0079] It should be understood that the attention mechanism is essentially a set of weight coefficients learned autonomously by the network, which are used in a "dynamically weighted" manner to emphasize areas of interest while suppressing irrelevant background areas. In the field of computer vision, attention mechanisms can be roughly divided into two categories: strong attention and soft attention. Because strong attention is a stochastic prediction that emphasizes dynamic changes, while effective, its non-differentiable nature limits its application. In contrast, soft attention is universally differentiable, meaning it can be achieved through neural network training based on gradient descent, resulting in a relatively wide range of applications.

[0080] Among them, temporal attention can be viewed as a dynamic temporal selection mechanism that determines when to pay attention, and is therefore commonly used in video processing. Convolutional neural networks and temporal pooling or weight learning have been widely used in video representation learning to capture interactions between frames, but these methods have limitations in efficiency or temporal relationship modeling. To overcome these problems, a global temporal representation is adopted to exploit multi-scale temporal cues in video sequences. The global temporal representation consists of a dilated temporal pyramid for learning local temporal context and a temporal self-attention module for capturing global temporal interactions. The dilated temporal pyramid uses dilated convolutions with gradually increasing dilation rates to cover different temporal ranges, and then concatenates the various outputs to aggregate multi-scale information.

[0081] More specifically, in the embodiments of the present application, Figure 3 FIG. 1 is a block diagram of the time attention encoding unit in the leg thermal therapy rehabilitation device according to an embodiment of the present application, as shown in FIG. Figure 3 As shown, the temporal attention encoding unit includes: an adjacent frame extraction unit 210, used to extract the adjacent first frame and second frame from the multiple thermotherapy monitoring reflected light component key frames; a first image local feature extraction unit 220, used to pass the first frame and the second frame respectively through the first convolution layer and the second convolution layer of the first convolutional neural network model using the temporal attention mechanism to obtain a first frame feature map corresponding to the first frame and a second frame feature map corresponding to the second frame; a temporal attention unit 230, used to perform positional point multiplication on the first frame feature map and the second frame feature map and then pass the Softmax activation function to obtain a temporal attention map; a second image local feature extraction unit 240, used to pass the second frame through the third convolution layer of the first convolutional neural network model using the temporal attention mechanism to obtain a second frame initial feature map; and a temporal attention applying unit 250, used to perform positional point multiplication on the second frame initial feature map and the temporal attention map to obtain a thermotherapy monitoring feature map.

[0082] The first convolutional neural network model of the temporal attention mechanism is used to perform feature extraction on each of the multiple thermotherapy monitoring reflected light component key frames, so as to extract the dynamic change feature information of the skin color of the patient's legs in the time dimension. The heating power of the thermotherapy rehabilitation equipment can be adaptively adjusted based on the changes in the skin state of the patient's legs during thermotherapy, so as to control the temperature of the thermotherapy and avoid secondary injuries to the patient caused by burns.

[0083] Specifically, in this embodiment of the present application, the dimension adjustment unit 150 is configured to perform global mean pooling on each feature matrix along the channel dimension of the thermotherapy monitoring feature map to obtain a thermotherapy monitoring feature vector. Subsequently, global mean pooling is performed on each feature matrix along the channel dimension of the thermotherapy monitoring feature map to retain characteristic information about temporal changes in skin color of the patient's legs in the thermotherapy monitoring feature map during data feature dimensionality reduction, thereby reducing computational complexity and preventing overfitting, thereby obtaining a thermotherapy monitoring feature vector.

[0084] Specifically, in an embodiment of the present application, the power feature extraction unit 160 is used to arrange the power values ​​of the multiple predetermined time points into a power input vector according to the time dimension and then pass it through a multi-scale neighborhood feature extraction module to obtain a multi-scale power feature vector. Furthermore, considering that the power value of the heater of the leg thermal therapy rehabilitation device has different pattern characteristics under different time period spans, therefore, in order to accurately extract the implicit change feature information of the power value of the heater, the power values ​​of the multiple predetermined time points are further arranged into a power input vector according to the time dimension and then encoded in a multi-scale neighborhood feature extraction module to extract the multi-scale neighborhood correlation features of the power value under different time spans within the predetermined time period, thereby obtaining a multi-scale power feature vector.

[0085] More specifically, in the embodiments of the present application, Figure 4 FIG. 1 is a block diagram of the power feature extraction unit in the leg hyperthermia rehabilitation device according to an embodiment of the present application, as shown in FIG. Figure 4As shown, the power feature extraction unit includes: a first-scale feature extraction unit 310, used to input the power input vector into the first convolution layer of the multi-scale neighborhood feature extraction module to obtain a first-scale power feature vector, wherein the first convolution layer has a first one-dimensional convolution kernel of a first length; a second-scale feature extraction unit 320, used to input the power input vector into the second convolution layer of the multi-scale neighborhood feature extraction module to obtain a second-scale power feature vector, wherein the second convolution layer has a second one-dimensional convolution kernel of a second length, and the first length is different from the second length; and a multi-scale cascade unit 330, used to cascade the first-scale power feature vector and the second-scale power feature vector to obtain the multi-scale power feature vector.

[0086] Furthermore, the first convolution layer of the multi-scale neighborhood feature extraction module is used to perform one-dimensional convolution encoding on the power input vector using the following formula to obtain the first-scale power feature vector; wherein the formula is:

[0087]

[0088] Wherein, a is the width of the first convolution kernel in the x direction, F(a) is the first convolution kernel parameter vector, G(xa) is the local vector matrix operated with the convolution kernel function, w is the size of the first convolution kernel, and X represents the power input vector.

[0089] Furthermore, the second convolution layer of the multi-scale neighborhood feature extraction module is used to perform one-dimensional convolution encoding on the power input vector using the following formula to obtain the second-scale power feature vector; wherein the formula is:

[0090]

[0091] Wherein, b is the width of the second convolution kernel in the x direction, F(b) is the second convolution kernel parameter vector, G(xb) is the local vector matrix operated with the convolution kernel function, m is the size of the second convolution kernel, and X represents the power input vector.

[0092] It should be understood that the power input vector is encoded by the multi-scale neighborhood feature extraction module to extract the multi-scale neighborhood correlation features of the power value at different time spans within the predetermined time period, so that the subsequent classification results are more accurate.

[0093] Specifically, in this embodiment of the present application, the responsiveness estimation unit 170 is configured to calculate a responsiveness estimate of the thermotherapy monitoring feature vector relative to the multi-scale power feature vector to obtain a classification feature matrix. The responsiveness estimate of the thermotherapy monitoring feature vector relative to the multi-scale power feature vector is then calculated to represent the correlation between the state change characteristics of the thermotherapy of the patient's leg and the dynamic change characteristics of the heating power of the thermotherapy device, and this is used to perform classification to obtain a classification result for adjusting the heating power value at the current time point.

[0094] Furthermore, Figure 5 is a block diagram of the responsiveness estimation unit in the leg thermal therapy rehabilitation device according to an embodiment of the present application, as shown in FIG. Figure 5 As shown, the responsiveness estimation unit includes: a Gaussian density map construction subunit 410, which is used to construct a self-Gaussian density map of the thermal therapy monitoring feature vector and the multi-scale power feature vector to obtain a first self-Gaussian density map and a second self-Gaussian density map; a responsiveness subunit 420, which is used to calculate the responsiveness estimation of the first self-Gaussian density map relative to the second self-Gaussian density map to obtain a responsiveness Gaussian density map; and a Gaussian discretization subunit 430, which is used to perform Gaussian discretization on the Gaussian distribution of each position of the responsiveness Gaussian density map to obtain the classification feature matrix.

[0095] Specifically, the correlation between the state change characteristics of the patient's leg thermal therapy and the dynamic change characteristics of the heating power of the thermal therapy device is represented by calculating the responsiveness estimate of the thermal therapy monitoring feature vector relative to the multi-scale power feature vector. This is then used to perform classification and obtain the adjusted classification result for the heating power value at the current time point. This enhances the accuracy of the responsiveness calculation between sequences, thereby improving the accuracy of the classification.

[0096] In particular, considering that the heating power value is volatile and uncertain, and the state change of the thermal therapy of the patient's legs will also be volatile and uncertain with the change of the heating power, in order to improve the accuracy of the power control of the heater at the current time point according to the actual situation of the patient's legs, it is necessary to perform data enhancement on the state change characteristics of the thermal therapy of the patient's legs and the dynamic change characteristics of the heating power of the thermal therapy equipment in a high-dimensional feature space.

[0097] It should be understood that as a learning target for a neural network model, a Gaussian density map can represent the joint distribution of a single eigenvalue of a feature distribution due to its probability density when multiple eigenvalues ​​constitute an overall distribution. That is, using the feature distribution as a prior distribution, the probability density at each prior distribution position due to the correlation of other prior distribution positions is obtained as a posterior distribution, thereby more accurately describing the feature distribution in a higher dimension. Therefore, in the technical solution of the present application, a self-Gaussian density map of the hyperthermia monitoring feature vector and the multi-scale power feature vector is constructed for data enhancement, thereby obtaining a first self-Gaussian density map and a second self-Gaussian density map.

[0098] Furthermore, the responsiveness estimate of the first self-Gaussian density map relative to the second self-Gaussian density map can be calculated to represent the correlation characteristics between the state change characteristics of the patient's leg thermal therapy and the dynamic change characteristics of the heating power of the thermal therapy device, thereby obtaining a responsive Gaussian density map. Next, the Gaussian distribution at each position of the responsive Gaussian density map is Gaussian discretized to avoid information loss during data feature augmentation, thereby obtaining the classification feature matrix. The classification feature matrix is ​​then used to perform classification processing in a classifier to obtain a classification result indicating whether the power value of the heater at the current time point should be increased or decreased.

[0099] Specifically, in this embodiment of the present application, the control result generation unit 180 is configured to pass the classification feature matrix through a classifier to obtain a classification result, which indicates whether the power value of the heater at the current time point should be increased or decreased. In this way, the heating power of the thermal therapy rehabilitation device can be adaptively adjusted based on the changes in the skin condition of the patient's leg during thermal therapy, thereby controlling the thermal therapy temperature and preventing the patient from being burned and causing secondary injury.

[0100] Furthermore, the classification feature matrix is ​​processed by the classifier using the following formula to generate a classification result; wherein the formula is: softmax{(M c , B c )|Project(F)}, where Project(F) represents projecting the classification feature matrix into a vector, M c is the weight matrix of the fully connected layer, B c Represents the bias vector of the fully connected layer.

[0101] In an embodiment of the present application, the leg thermal therapy rehabilitation device also includes a training module for training the first convolutional neural network model using the temporal attention mechanism, the multi-scale neighborhood feature extraction module and the classifier. Figure 6 is a block diagram of the training module in the leg thermal therapy rehabilitation device according to an embodiment of the present application, as shown in FIG. Figure 6 As shown, the training module 500 includes: a training data acquisition unit 501, which is used to acquire training data, wherein the training data includes the training power value of the leg thermotherapy rehabilitation device at multiple predetermined time points within a predetermined time period, the leg thermotherapy training monitoring video of the predetermined time period, and the real value of the power value of the heater at the current time point that should be increased or decreased; a training sampling unit 502, which is used to extract multiple training thermotherapy monitoring key frames from the leg thermotherapy training monitoring video; a training reflected light component extraction unit 503, which is used to extract the reflected light component of each pixel of each training thermotherapy monitoring key frame in the multiple training thermotherapy monitoring key frames to obtain multiple training thermotherapy monitoring reflected light component key frames; a training time attention encoding unit 504, which is used to pass the multiple training thermotherapy monitoring reflected light component key frames through the first convolutional neural network model using the time attention mechanism to obtain a training thermotherapy monitoring feature map; a training dimension adjustment unit 505, which is used to perform global mean pooling on each feature matrix along the channel dimension of the training thermotherapy monitoring feature map to obtain a training thermotherapy monitoring key frame. monitoring feature vector; a training power feature extraction unit 506, for arranging the training power values ​​of the multiple predetermined time points into a training power input vector according to the time dimension and then passing it through the multi-scale neighborhood feature extraction module to obtain a training multi-scale power feature vector; a training responsiveness estimation unit 507, for calculating the responsiveness estimation of the training hyperthermia monitoring feature vector relative to the training multi-scale power feature vector to obtain a training classification feature matrix; a classification loss unit 508, for passing the training classification feature matrix through the classifier to obtain a classification loss function value; an internalized learning loss unit 509, for calculating the sequence-to-sequence response rule internalized learning loss function value based on the distance between the training hyperthermia monitoring feature vector and the training multi-scale power feature vector; and a training unit 510, for calculating the weighted sum of the classification loss function value and the sequence-to-sequence response rule internalized learning loss function value as a loss function value to train the first convolutional neural network model using the temporal attention mechanism, the multi-scale neighborhood feature extraction module and the classifier.

[0102] In particular, in the technical solution of the present application, by calculating the responsiveness estimate of the thermal therapy monitoring feature vector relative to the multi-scale power feature vector based on the Gaussian density map, the obtained classification feature matrix can reflect the responsiveness relationship between the overall feature probability density distribution of the thermal therapy monitoring feature vector and the multi-scale power feature vector.

[0103] Preferably, in order to further enhance the accuracy of the responsiveness calculation between the thermotherapy monitoring feature vector and the multi-scale power feature vector, the eigenvalue sequence distribution characteristics of the thermotherapy monitoring feature vector and the multi-scale power feature vector can be used to learn the cause-effect response between the sequences under a predetermined classification task.

[0104] Specifically, a sequence-to-sequence response rule internalization learning loss function is introduced. That is, the sequence-to-sequence response rule internalization learning loss function value is calculated based on the distance between the training hyperthermia monitoring feature vector and the training multi-scale power feature vector using the following formula; wherein, the formula is:

[0105]

[0106]

[0107]

[0108] Wherein, V1 is the training hyperthermia monitoring feature vector, V2 is the training multi-scale power feature vector, and W1 and W2 are the weight matrices of the classifier for the training hyperthermia monitoring feature vector and the training multi-scale power feature vector, respectively. ReLU(·) represents the ReLU activation function, and Sigmoid(·) represents the Sigmoid activation function. represents matrix multiplication, and d(·,·) represents the Euclidean distance between two vectors.

[0109] Here, the classifier uses a squeeze-and-excite channel attention mechanism for the weight matrices of different sequences to obtain enhanced discriminative capabilities between sequences. By training the network with this loss function, it is possible to restore more discriminative causal features between response sequences, thereby internalizing the cause-and-effect response rules between sequences. This enhances the accuracy of responsiveness calculations between sequences and improves classification accuracy. In this way, the heating power of the thermal therapy rehabilitation device can be adaptively adjusted based on the changes in the skin condition of the patient's leg during thermal therapy, thereby controlling the temperature of the thermal therapy and preventing the patient from being burned and causing secondary injuries.

[0110] In summary, the leg thermotherapy rehabilitation device 100 based on the embodiment of the present application is explained, which mines the hidden features of the image of the patient's leg thermotherapy monitoring video in a predetermined time period by using the first convolutional neural network model of the time attention mechanism to extract the state change feature information of the patient's leg skin color in the time dimension; extracts the multi-scale neighborhood correlation features of the power value of the heater of the leg thermotherapy rehabilitation device under different time spans within the predetermined time period through the multi-scale neighborhood feature extraction module; then, based on the state change characteristics of the patient's leg thermotherapy, the heating power of the heater of the thermotherapy rehabilitation device is adaptively adjusted, and the thermotherapy temperature is dynamically controlled in real time based on the actual situation. In this way, the temperature of the thermotherapy can be intelligently controlled to avoid secondary injuries caused by burns to the patient.

[0111] Exemplary Methods

[0112] Figure 7 FIG. 1 is a flow chart of a method for using a leg hyperthermia rehabilitation device according to an embodiment of the present application. Figure 7 As shown, the method for using the leg thermotherapy rehabilitation device according to the embodiment of the present application includes: S110, obtaining the power value of the heater of the leg thermotherapy rehabilitation device at multiple predetermined time points within a predetermined time period and the leg thermotherapy monitoring video of the predetermined time period collected by the camera; S120, extracting multiple thermotherapy monitoring key frames from the leg thermotherapy monitoring video; S130, extracting the reflected light component of each pixel of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames to obtain multiple thermotherapy monitoring reflected light component key frames; S140, obtaining the multiple thermotherapy monitoring reflected light component key frames by using the first convolutional neural network model of the time attention mechanism A thermotherapy monitoring feature map; S150, performing global mean pooling on each feature matrix along the channel dimension of the thermotherapy monitoring feature map to obtain a thermotherapy monitoring feature vector; S160, arranging the power values ​​of the multiple predetermined time points into a power input vector according to the time dimension and then passing it through a multi-scale neighborhood feature extraction module to obtain a multi-scale power feature vector; S170, calculating the responsiveness estimate of the thermotherapy monitoring feature vector relative to the multi-scale power feature vector to obtain a classification feature matrix; and, S180, passing the classification feature matrix through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the power value of the heater at the current time point should be increased or decreased.

[0113] Figure 8 Schematic diagram of the system architecture of the method for using the leg thermal therapy rehabilitation device according to an embodiment of the present application. Figure 8As shown, in the system architecture of the method for using the leg thermotherapy rehabilitation device, first, the power value of the heater of the leg thermotherapy rehabilitation device at multiple predetermined time points within a predetermined time period and the leg thermotherapy monitoring video of the predetermined time period collected by the camera are obtained; then, multiple thermotherapy monitoring key frames are extracted from the leg thermotherapy monitoring video; then, the reflected light component of each pixel of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames is extracted to obtain multiple thermotherapy monitoring reflected light component key frames; then, the multiple thermotherapy monitoring reflected light component key frames are used to obtain the thermotherapy monitoring feature map; then, performing global mean pooling on each feature matrix along the channel dimension of the thermotherapy monitoring feature map to obtain a thermotherapy monitoring feature vector; then, arranging the power values ​​of the multiple predetermined time points into a power input vector according to the time dimension and passing it through a multi-scale neighborhood feature extraction module to obtain a multi-scale power feature vector; then, calculating the responsiveness estimate of the thermotherapy monitoring feature vector relative to the multi-scale power feature vector to obtain a classification feature matrix; and, finally, passing the classification feature matrix through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the power value of the heater at the current time point should be increased or decreased.

[0114] In a specific example, in the method for using the above-mentioned leg thermotherapy rehabilitation device, extracting multiple thermotherapy monitoring key frames from the leg thermotherapy monitoring video further includes: extracting the multiple thermotherapy monitoring key frames from the leg thermotherapy monitoring video at a predetermined sampling frequency.

[0115] In a specific example, in the method for using the above-mentioned leg thermotherapy rehabilitation device, extracting the reflected light component of each pixel of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames to obtain multiple thermotherapy monitoring reflected light component key frames further includes: extracting the reflected light component of each pixel of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames using the following formula to obtain the multiple thermotherapy monitoring reflected light component key frames; wherein the formula is:

[0116] S(x, y) = R(x, y) L(x, y)

[0117] Wherein, S(x, y) represents each pixel value of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames, R(x, y) represents each pixel of each thermotherapy monitoring reflected light component key frame in the multiple thermotherapy monitoring reflected light component key frames, L(x, y) represents each pixel of each thermotherapy monitoring brightness component key frame in the multiple thermotherapy monitoring brightness component key frames, and the symbol · represents matrix multiplication.

[0118] In a specific example, in the method of using the above-mentioned leg thermotherapy rehabilitation equipment, the multiple thermotherapy monitoring reflected light component key frames are passed through the first convolutional neural network model using the temporal attention mechanism to obtain a thermotherapy monitoring feature map, including: extracting adjacent first and second frames from the multiple thermotherapy monitoring reflected light component key frames; passing the first frame and the second frame through the first convolutional layer and the second convolutional layer of the first convolutional neural network model using the temporal attention mechanism respectively to obtain a first frame feature map corresponding to the first frame and a second frame feature map corresponding to the second frame; performing positional dot multiplication on the first frame feature map and the second frame feature map and then passing through the Softmax activation function to obtain a temporal attention map; passing the second frame through the third convolutional layer of the first convolutional neural network model using the temporal attention mechanism to obtain a second frame initial feature map; and performing positional dot multiplication on the second frame initial feature map and the temporal attention map to obtain a thermotherapy monitoring feature map.

[0119] In a specific example, in the method of using the above-mentioned leg thermal therapy rehabilitation device, the power values ​​of the multiple predetermined time points are arranged as a power input vector according to the time dimension and then passed through a multi-scale neighborhood feature extraction module to obtain a multi-scale power feature vector, including: inputting the power input vector into the first convolution layer of the multi-scale neighborhood feature extraction module to obtain a first-scale power feature vector, wherein the first convolution layer has a first one-dimensional convolution kernel of a first length; inputting the power input vector into the second convolution layer of the multi-scale neighborhood feature extraction module to obtain a second-scale power feature vector, wherein the second convolution layer has a second one-dimensional convolution kernel of a second length, and the first length is different from the second length; and, cascading the first-scale power feature vector and the second-scale power feature vector to obtain the multi-scale power feature vector.

[0120] In a specific example, in the method for using the above-mentioned leg thermal therapy rehabilitation device, the power input vector is input into the first convolution layer of the multi-scale neighborhood feature extraction module to obtain a first-scale power feature vector, wherein the first convolution layer has a first one-dimensional convolution kernel of a first length, further comprising: using the first convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the power input vector using the following formula to obtain the first-scale power feature vector; wherein the formula is:

[0121]

[0122] Wherein, a is the width of the first convolution kernel in the x direction, F(a) is the first convolution kernel parameter vector, G(xa) is the local vector matrix operated with the convolution kernel function, w is the size of the first convolution kernel, and X represents the power input vector.

[0123] Inputting the power input vector into the second convolution layer of the multi-scale neighborhood feature extraction module to obtain a second-scale power feature vector, wherein the second convolution layer has a second one-dimensional convolution kernel of a second length, and the first length is different from the second length, further comprising: using the second convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the power input vector using the following formula to obtain the second-scale power feature vector; wherein the formula is:

[0124]

[0125] Wherein, b is the width of the second convolution kernel in the x direction, F(b) is the second convolution kernel parameter vector, G(xb) is the local vector matrix operated with the convolution kernel function, m is the size of the second convolution kernel, and X represents the power input vector.

[0126] In a specific example, in the method of using the above-mentioned leg thermal therapy rehabilitation device, the calculation of the responsiveness estimate of the thermal therapy monitoring feature vector relative to the multi-scale power feature vector to obtain a classification feature matrix includes: constructing a self-Gaussian density map of the thermal therapy monitoring feature vector and the multi-scale power feature vector to obtain a first self-Gaussian density map and a second self-Gaussian density map; calculating the responsiveness estimate of the first self-Gaussian density map relative to the second self-Gaussian density map to obtain a responsive Gaussian density map; and performing Gaussian discretization on the Gaussian distribution of each position of the responsive Gaussian density map to obtain the classification feature matrix.

[0127] In a specific example, in the method for using the above-mentioned leg thermal therapy rehabilitation device, the classification feature matrix is ​​passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the power value of the heater at the current time point should be increased or decreased, further comprising: using the classifier to process the classification feature matrix using the following formula to generate a classification result; wherein the formula is: softmax{(M c , B c )|Project(F)}, where Project(F) represents projecting the classification feature matrix into a vector, M c is the weight matrix of the fully connected layer, B c Represents the bias vector of the fully connected layer.

[0128] In a specific example, the method for using the above-mentioned leg thermotherapy rehabilitation device also includes: training the first convolutional neural network model using the time attention mechanism, the multi-scale neighborhood feature extraction module and the classifier; wherein, the training of the first convolutional neural network model using the time attention mechanism, the multi-scale neighborhood feature extraction module and the classifier includes: obtaining training data, the training data including the training power value of the leg thermotherapy rehabilitation device at multiple predetermined time points within a predetermined time period, the leg thermotherapy training monitoring video of the predetermined time period, and the actual value of the power value of the heater at the current time point that should be increased or decreased; extracting multiple training thermotherapy monitoring key frames from the leg thermotherapy training monitoring video; extracting the reflected light component of each pixel of each training thermotherapy monitoring key frame in the multiple training thermotherapy monitoring key frames to obtain multiple training thermotherapy monitoring reflected light component key frames; passing the multiple training thermotherapy monitoring reflected light component key frames through the first convolutional neural network model using the time attention mechanism. The invention relates to a first convolutional neural network model using a temporal attention mechanism, wherein the first convolutional neural network model is used to obtain a training hyperthermia monitoring feature map; global mean pooling is performed on each feature matrix of the training hyperthermia monitoring feature map along the channel dimension to obtain a training hyperthermia monitoring feature vector; the training power values ​​of the multiple predetermined time points are arranged according to the time dimension as a training power input vector and then passed through the multi-scale neighborhood feature extraction module to obtain a training multi-scale power feature vector; the responsiveness estimate of the training hyperthermia monitoring feature vector relative to the training multi-scale power feature vector is calculated to obtain a training classification feature matrix; the training classification feature matrix is ​​passed through the classifier to obtain a classification loss function value; the sequence-to-sequence response rule internalization learning loss function value is calculated based on the distance between the training hyperthermia monitoring feature vector and the training multi-scale power feature vector; and the weighted sum of the classification loss function value and the sequence-to-sequence response rule internalization learning loss function value is calculated as the loss function value to train the first convolutional neural network model using the temporal attention mechanism, the multi-scale neighborhood feature extraction module and the classifier.

[0129] In a specific example, in the method for using the above-mentioned leg hyperthermia rehabilitation device, the calculation of the sequence-to-sequence response rule internalization learning loss function value based on the distance between the training hyperthermia monitoring feature vector and the training multi-scale power feature vector further includes: calculating the sequence-to-sequence response rule internalization learning loss function value based on the distance between the training hyperthermia monitoring feature vector and the training multi-scale power feature vector using the following formula; wherein the formula is:

[0130]

[0131]

[0132]

[0133] Wherein, V1 is the training hyperthermia monitoring feature vector, V2 is the training multi-scale power feature vector, and W1 and W2 are the weight matrices of the classifier for the training hyperthermia monitoring feature vector and the training multi-scale power feature vector, respectively. ReLU(·) represents the ReLU activation function, and Sigmoid(·) represents the Sigmoid activation function. represents matrix multiplication, and d(·,·) represents the Euclidean distance between two vectors.

[0134] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned method of using the leg thermal therapy rehabilitation device have been described in the above reference. Figures 1 to 6 The description of the leg thermal therapy rehabilitation device has been introduced in detail, and therefore, its repeated description will be omitted.

Claims

1. A leg hyperthermia rehabilitation device, characterized in that: include: a data monitoring unit, configured to obtain power values ​​of a heater of a leg hyperthermia rehabilitation device at a plurality of predetermined time points within a predetermined time period and a monitoring video of the leg hyperthermia during the predetermined time period captured by a camera; a sampling unit, configured to extract a plurality of hyperthermia monitoring key frames from the leg hyperthermia monitoring video; a reflected light component extraction unit, configured to extract the reflected light component of each pixel of each of the plurality of hyperthermia monitoring key frames to obtain a plurality of hyperthermia monitoring reflected light component key frames; A temporal attention encoding unit is configured to obtain a thermotherapy monitoring feature map by encoding the plurality of thermotherapy monitoring reflected light component key frames through a first convolutional neural network model using a temporal attention mechanism; a dimension adjustment unit, configured to perform global mean pooling on each feature matrix along the channel dimension of the hyperthermia monitoring feature map to obtain a hyperthermia monitoring feature vector; A power feature extraction unit, configured to arrange the power values ​​of the plurality of predetermined time points into a power input vector according to the time dimension and then obtain a multi-scale power feature vector by a multi-scale neighborhood feature extraction module; a responsiveness estimation unit, configured to calculate a responsiveness estimate of the hyperthermia monitoring feature vector relative to the multi-scale power feature vector to obtain a classification feature matrix; as well as a control result generating unit, configured to pass the classification feature matrix through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the power value of the heater at a current time point should be increased or decreased; Also included is a training module for training the first convolutional neural network model using the temporal attention mechanism, the multi-scale neighborhood feature extraction module, and the classifier; Wherein, the training module includes: a training data acquisition unit, configured to acquire training data, the training data comprising training power values ​​of the leg thermotherapy rehabilitation device at a plurality of predetermined time points within a predetermined time period, a monitoring video of the leg thermotherapy training during the predetermined time period, and a real value at which the power value of the heater should be increased or decreased at the current time point; a training sampling unit, configured to extract a plurality of training hyperthermia monitoring key frames from the leg hyperthermia training monitoring video; a training reflected light component extraction unit, configured to extract the reflected light component of each pixel of each training hyperthermia monitoring key frame in the plurality of training hyperthermia monitoring key frames to obtain a plurality of training hyperthermia monitoring reflected light component key frames; a training temporal attention encoding unit, configured to pass the plurality of training hyperthermia monitoring reflected light component key frames through the first convolutional neural network model using the temporal attention mechanism to obtain a training hyperthermia monitoring feature map; a training dimension adjustment unit, configured to perform global mean pooling on each feature matrix along the channel dimension of the training hyperthermia monitoring feature map to obtain a training hyperthermia monitoring feature vector; a training power feature extraction unit, configured to arrange the training power values ​​of the plurality of predetermined time points into a training power input vector according to the time dimension and then pass the vector through the multi-scale neighborhood feature extraction module to obtain a training multi-scale power feature vector; a training responsiveness estimation unit, configured to calculate a responsiveness estimate of the training hyperthermia monitoring feature vector relative to the training multi-scale power feature vector to obtain a training classification feature matrix; a classification loss unit, configured to pass the training classification feature matrix through the classifier to obtain a classification loss function value; an internalized learning loss unit, configured to calculate a sequence-to-sequence response rule internalized learning loss function value based on a distance between the training hyperthermia monitoring feature vector and the training multi-scale power feature vector; and A training unit is used to calculate the weighted sum of the classification loss function value and the sequence-to-sequence response rule internalization learning loss function value as the loss function value to train the first convolutional neural network model using the temporal attention mechanism, the multi-scale neighborhood feature extraction module and the classifier.

2. The leg hyperthermia rehabilitation device according to claim 1, characterized in that: The sampling unit is further used to extract the multiple thermotherapy monitoring key frames from the leg thermotherapy monitoring video at a predetermined sampling frequency.

3. The leg hyperthermia rehabilitation device according to claim 2, characterized in that: The reflected light component extraction unit is further configured to extract the reflected light component of each pixel of each of the multiple hyperthermia monitoring key frames using the following formula to obtain multiple hyperthermia monitoring reflected light component key frames; Wherein, the formula is: S(x,y)=R(x,y)·L(x,y) Wherein, S(x,y) represents each pixel value of each thermotherapy monitoring key frame in the multiple thermotherapy monitoring key frames, R(x,y) represents each pixel of each thermotherapy monitoring reflected light component key frame in the multiple thermotherapy monitoring reflected light component key frames, L(x,y) represents each pixel of each thermotherapy monitoring brightness component key frame in the multiple thermotherapy monitoring brightness component key frames, and the symbol · represents matrix multiplication.

4. The leg hyperthermia rehabilitation device according to claim 3, characterized in that: The temporal attention encoding unit comprises: an adjacent frame extraction unit, configured to extract an adjacent first frame and a second frame from the plurality of thermotherapy monitoring reflected light component key frames; A first image local feature extraction unit is configured to pass the first frame and the second frame through the first convolutional layer and the second convolutional layer of the first convolutional neural network model using the temporal attention mechanism, respectively, to obtain a first frame feature map corresponding to the first frame and a second frame feature map corresponding to the second frame; A temporal attention unit, configured to perform a positional dot multiplication on the first frame feature map and the second frame feature map and then pass the result through a Softmax activation function to obtain a temporal attention map; A second image local feature extraction unit is configured to pass the second frame through the third convolutional layer of the first convolutional neural network model using the temporal attention mechanism to obtain an initial feature map of the second frame; and A temporal attention applying unit is used to multiply the second frame initial feature map by the temporal attention map by position points to obtain a hyperthermia monitoring feature map.

5. The leg hyperthermia rehabilitation device according to claim 4, characterized in that: The power feature extraction unit includes: a first scale feature extraction unit, configured to input the power input vector into a first convolutional layer of the multi-scale neighborhood feature extraction module to obtain a first scale power feature vector, wherein the first convolutional layer has a first one-dimensional convolution kernel of a first length; a second-scale feature extraction unit, configured to input the power input vector into a second convolutional layer of the multi-scale neighborhood feature extraction module to obtain a second-scale power feature vector, wherein the second convolutional layer has a second one-dimensional convolution kernel of a second length, and the first length is different from the second length; and A multi-scale cascading unit is configured to cascade the first-scale power feature vector and the second-scale power feature vector to obtain the multi-scale power feature vector.

6. The leg thermal therapy rehabilitation device according to claim 5, characterized in that: The first-scale feature extraction unit is further configured to: use the first convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the power input vector using the following formula to obtain the first-scale power feature vector; Wherein, the formula is: Wherein, a is the width of the first convolution kernel in the x direction, F(a) is the first convolution kernel parameter vector, G(xa) is the local vector matrix operated with the convolution kernel function, w is the size of the first convolution kernel, and X represents the power input vector; The second-scale feature extraction unit is further configured to: use the second convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the power input vector using the following formula to obtain the second-scale power feature vector; Wherein, the formula is: Wherein, b is the width of the second convolution kernel in the x direction, F(b) is the second convolution kernel parameter vector, G(xb) is the local vector matrix operated with the convolution kernel function, m is the size of the second convolution kernel, and X represents the power input vector.

7. The leg thermal therapy rehabilitation device according to claim 6, characterized in that: The responsiveness estimation unit comprises: a Gaussian density map construction subunit, configured to construct a self-Gaussian density map of the hyperthermia monitoring feature vector and the multi-scale power feature vector to obtain a first self-Gaussian density map and a second self-Gaussian density map; a responsiveness subunit, configured to calculate a responsiveness estimate of the first self-Gaussian density map relative to the second self-Gaussian density map to obtain a responsiveness Gaussian density map; and The Gaussian discretization subunit is used to perform Gaussian discretization on the Gaussian distribution of each position of the responsive Gaussian density map to obtain the classification feature matrix.

8. The leg thermal therapy rehabilitation device according to claim 7, characterized in that: The control result generating unit is further used to: use the classifier to process the classification feature matrix according to the following formula to generate a classification result; wherein the formula is: softmax{(M c ,B c )|Project(F)}, where Project(F) represents projecting the classification feature matrix into a vector, M c is the weight matrix of the fully connected layer, B c Represents the bias vector of the fully connected layer.

9. The leg hyperthermia rehabilitation device according to claim 1, characterized in that: The internalized learning loss unit is further configured to calculate the sequence-to-sequence response rule internalized learning loss function value using the following formula based on the distance between the training hyperthermia monitoring feature vector and the training multi-scale power feature vector; Wherein, the formula is: Wherein, V1 is the training hyperthermia monitoring feature vector, V2 is the training multi-scale power feature vector, and W1 and W2 are the weight matrices of the classifier for the training hyperthermia monitoring feature vector and the training multi-scale power feature vector, respectively. ReLU(·) represents the ReLU activation function, and Sigmoid(·) represents the Sigmoid activation function. represents matrix multiplication, and d(·,·) represents the Euclidean distance between two vectors.

Images