A target identity detection system and method for a machine-assisted rehabilitation system

Abstract

The application discloses a target identity detection system and method for a machine-assisted rehabilitation system, and comprises an RSSD module, a TA-LSTM module, a target detection module and a real-time identity recognition module, wherein the RSSD module is used for extracting patient limb or auxiliary tool detection box information in each frame of the rehabilitation video; the TA-LSTM module is used for detecting the target, suppressing useless features to reduce the calculation amount, and enhancing the target features; the target detection module is used for detecting the patient limb or auxiliary tool detection box information in each frame of the rehabilitation video, and the detection box information comprises a target category score, a detection box center point coordinate and a width and height; and the real-time identity recognition module is used for performing identity recognition on the target of the dynamic frame and performing corresponding identity labeling. In the application, the joint training of the static frame and the video stream is used to realize inter-frame mining, so that the detection efficiency is improved, and the false detection and missed detection of the problem frame caused by patient movement, abnormal monitoring camera and the like are avoided.

Description

Technical Field

[0001] This invention relates to the field of target detection and identity recognition technology in video, specifically a target identity detection system and method for machine-assisted rehabilitation systems. Background Technology

[0002] To break away from the fixed pattern of traditional rehabilitation systems that only maintain patients in a prescribed exercise state, Fudan University proposed an intelligent rehabilitation robot data processing method and system for assisting rehabilitation training (patent publication number: CN114099234A). This invention preprocesses and analyzes acquired limb image data and constructs a limb data analysis model based on this. The limb analysis model is then used to analyze the current state and behavior of people with disabilities and to develop corresponding personalized rehabilitation training programs, thereby assisting them in their rehabilitation training. Although this invention breaks the fixed pattern of traditional rehabilitation systems, some remaining issues still need to be addressed.

[0003] First, the video target detection data analysis of traditional rehabilitation systems is mainly based on static frames and does not involve mining the inter-frame relationships. Therefore, its performance needs to be improved. For videos, there are obvious temporal context relationships between targets in adjacent frames, which can solve the following problems: there is a lot of redundant information between adjacent frames, and the images are blurred, distorted, or occluded due to patient movement, environmental interference, and camera malfunctions.

[0004] Secondly, traditional rehabilitation systems' video object detection hasn't reached the level of small object detection. This leads to patients using different tools to assist in training for different needs, such as grasping small objects. Furthermore, due to the visual difference of near objects appearing larger than far objects, patients performing rehabilitation training at a distance from the camera, or gradually moving away from the camera during training due to the designed movements, result in the detected object occupying too small a proportion of the detection frame. Because small objects have too few visible features and are difficult to match with candidate boxes, conventional object detection methods perform poorly, thus limiting the proportion of the detected object in the detection frame.

[0005] Third, traditional rehabilitation systems do not optimize video target detection for complex backgrounds, i.e., foreground extraction in complex backgrounds. If rehabilitation systems are to better serve patients, they need to be integrated into various hospital settings and even patients' home life. If the detection background is complex, it will interfere with the detection results, thus limiting the location of the rehabilitation system.

[0006] Fourth, traditional rehabilitation systems lack optimization for video target detection to handle scale changes. During rehabilitation training, the scale of targets changes drastically and irregularly. Most current target detection models are only suitable for features without scale changes, and their detection accuracy decreases for features with large scale changes.

[0007] Fifth, video target detection in traditional rehabilitation systems does not involve object identification, which can easily lead to the following problems: similar body parts of individual patients can generate similar objects; multiple people entering the detection range can generate multiple similar body parts; and patients using multiple similar assistive devices can generate multiple similar objects, thus causing the rehabilitation system to make judgment errors. Summary of the Invention

[0008] The purpose of this invention is to provide a target identity detection system and method for machine-assisted rehabilitation systems, so as to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a target identity detection system for a machine-assisted rehabilitation system, comprising an RSSD module, a TA-LSTM module, a target detection module, and a real-time identity recognition module, characterized in that...

[0010] The RSSD module is used to extract the detection box information of the patient's limbs or assistive tools in each frame of the rehabilitation video.

[0011] The TA-LSTM module detects targets, suppresses useless features to reduce computation, and enhances target features.

[0012] The target detection module detects bounding box information for the patient's limbs or assistive tools in each frame of the rehabilitation video. The bounding box information includes the target category score, the coordinates of the center point of the bounding box, and its width and height.

[0013] The real-time identity recognition module identifies the target in the dynamic frame and assigns a corresponding identity number.

[0014] Preferably, the RSSD module replaces the VGG-16 backbone network of the SSD module with the Resnet-101 network, and combines the idea of ​​feature pyramid to obtain a three-level feature extraction structure to integrate shallow and deep features for subsequent modeling.

[0015] Preferably, in the RSSD module architecture, the number of channels in Conv4 of the RSSD backbone network is 1024; the number of channels in Conv8 of the last four layers of RSSD is 512. Based on the idea of ​​feature pyramid, Conv4 and all subsequent convolutional layers are divided into three parts: low-level feature map part, intermediate-level feature map part, and deep-level feature map part.

[0016] Preferably, the TA-LSTM module introduces the spatiotemporal sequence model ST-LSTM to realize inter-frame mining. Combined with the forget gate and correlation gate in the attention mechanism, a long-short time TA-LSTM module with a temporal attention mechanism is obtained. The attention mechanism provides the spatiotemporal sequence model with accurate object location regions to selectively update the memory. At the same time, the spatiotemporal sequence model provides rich temporal information for the attention process to generate a more accurate attention map. Thus, useless features are accurately suppressed and the amount of computation is reduced, thereby enhancing the target features, where useless features are redundant scale features and background features.

[0017] Preferably, the target detection module utilizes the SSD's Default Box mechanism to obtain the detection box information of the patient's limbs or assistive tools in each frame of the rehabilitation video. The detection box information includes the target category score, the coordinates of the detection box center point, and its width and height.

[0018] A target detection method for a machine-assisted rehabilitation system is characterized by: detecting and assigning unique IDs to patient limbs and assistive tools in rehabilitation videos; applying the trained model to the rehabilitation system using a joint training scheme; and the specific implementation steps are as follows:

[0019] Step 1: Use the RSSD module to extract features from static frames in the rehabilitation video;

[0020] Step 2: Use the TA-LSTM module to implement feature enhancement; the TA-LSTM method solves the real-time differentiation problem by representing the differences between different objects based on the attention map, extracting information from shallow features and reducing the extraction vector of shallow features to reduce computational cost;

[0021] Step 3: Use object detection algorithms to generate candidate bounding boxes for patient limbs or assistive devices in the rehabilitation video;

[0022] Step 4: Use a real-time identity recognition module to perform real-time identity recognition of the patient's limbs or assistive devices in the rehabilitation video.

[0023] Preferably, the TA-LSTM method for real-time differentiation uses attention maps to represent the differences between different objects, extracts information from shallow features, and reduces the extraction vector of shallow features to lower computational costs.

[0024] Preferably, the method of the real-time identity recognition module is to achieve real-time identity recognition of multiple objects through online Tubelet analysis. The input of detection box information in each frame is combined with the input of attention map output by low-level TA-LSTM to help distinguish similar targets in the video. Then, the non-maximum suppression idea is combined to suppress redundant detection boxes and extract the final detection box. Finally, according to the algorithm, a unique and continuous corresponding identity number is assigned to the patient's limbs or assistive tools in the rehabilitation video.

[0025] Preferably, the formation of a Tubelet from the attention map output by the low-level TA-LSTM and the detection box output by the detection network involves the following steps:

[0026] S1, Traverse the detected categories in the video;

[0027] S1.1 Determine if there is a detection box for this category. If yes, proceed to S1.2; otherwise, proceed to S1.4.

[0028] S1.2, Traverse the objects of this category that have detection boxes;

[0029] S1.2.1, Maximum correlation between the object and the detection box. The initial value is set to 0;

[0030] S1.2.2, Traverse all detection boxes that detect this category;

[0031] S1.2.3 Determine the correlation C between the detection box and the object. obj,tube Is it greater than If yes, proceed to S1.2.4; otherwise, return to S1.2.2 and continue iterating.

[0032] S1.2.4, C obj,tube Assign to At the same time, the value of the detection box is assigned to the candidate box CAND of the object;

[0033] S1.2.5, Judgment If the threshold tMT is greater than the threshold, proceed to S1.2.6; otherwise, return to S1.2 to continue the iteration.

[0034] S1.2.6, assign the candidate identity obj to this object. CAND[ID Assign the value to the object and return to S1.2 to continue the traversal. If the traversal ends, proceed to S1.3.

[0035] S1.3, Traverse all detection boxes that detect this category;

[0036] S1.3.1 Determine whether the detection box matches an object. If it does, proceed to S1.3.2; otherwise, return to S1.3 to continue iterating.

[0037] S1.3.2 Obtain the maximum correlation C of this category, and return to S1.3 to continue traversing. If the traversal ends, proceed to S1.4.

[0038] S1.4, Traverse objects that do not match the identity;

[0039] S1.4.1 Determine if the confidence level of the candidate number of the object is greater than the confidence level of the detection box. If so, proceed to S1.4.2; otherwise, return to S1.4 to continue traversing.

[0040] S1.4.2, the candidate body obj CAND[ID] Assign the value to the object and return to S1.4 to continue the traversal. If the traversal ends, proceed to S1.5.

[0041] S1.5 Update the detection box for this category and return to S1 to continue the traversal. If the traversal ends, proceed to S2.

[0042] S2, Output the identity of the object.

[0043] Preferably, the joint training method for static frames and videos involves the TA-LSTM model generating similar global memories for each frame. When a static frame with occlusion of detection box information or low image quality is detected, the TA-LSTM model can recall features from previous frames to assist in the detection of the current frame. The training method includes: 1) confidence loss L c ;2) Position loss L l 3) Attention deficit at ;4) Related loss L as The training process is performed only in RSSD and TA-LSTM, and the identity-aware algorithm does not participate in the training. The overall training calculation formula is as follows.

[0044] L=αL c,l +δL at +σL as (1)

[0045] In the formula, α, δ, and σ are the weights of each loss;

[0046] L c Training is performed using the Softmax loss function, L l Training is performed using a smoothed L1 loss function, and the joint loss of the two is L. c，l The calculation formula is:

[0047]

[0048] In the formula, N is the number of matched truth boxes, c represents the confidence score, d represents the default box, and g represents the truth box; x i，j Indicates whether the default box and the actual box match (x i，j =0 indicates no match, x i，j =1 indicates a match); β is used to adjust L c and L l The ratio, initially set to 1;

[0049] Each attention loss is trained using the cross-entropy function, and the multi-scale attention losses are summed to obtain the final loss L. at The calculation formula is:

[0050]

[0051] In the formula, AM g Represents the true attention map. This represents the k-th level attention map for prediction.

[0052] Since some static score calculations can lead to significant biases in the overall sequence during training, and improper correlations between individual local frames and the global sequence can also affect detection results, this invention proposes a self-supervised training method to mitigate pixel-level variations in video detection tasks and maintain temporal consistency across multiple frames. The key idea is to create a list of locally correlated frame scores (SL) to compute the correlation loss function L. as The calculation formula is:

[0053]

[0054] In the formula, SL t A list of fractions representing time t. S_LEN represents the average score from time 1 to time t-1, and S_LEN represents the length of the selected sequence.

[0055] Compared with the prior art, the beneficial effects of the present invention are:

[0056] 1. The target identity detection system and method for machine-assisted rehabilitation system replaces the traditional SSD backbone network with a Resnet-101 network and constructs a three-level feature extraction module RSSD module by combining the idea of ​​feature pyramid. This reduces the amount of computation and the time for extracting representative features from rehabilitation videos, and enables accurate detection of patients during remote training or training with small objects.

[0057] 2. The target identity detection system and method for machine-assisted rehabilitation system achieves inter-frame mining through joint training of static frames and video streams, thereby improving detection efficiency and avoiding false detections and missed detections of problematic frames caused by patient movement, abnormal monitoring cameras, etc.

[0058] 3. The target identity detection system and method for machine-assisted rehabilitation system redesigns the ST-LSTM network by combining the attention mechanism to construct the TA-LSTM network. This network achieves target detection in complex backgrounds and under conditions of large patient movements with less computation.

[0059] 4. The target identity detection system and method for machine-assisted rehabilitation system introduces a target identity recognition algorithm based on Tubelet and uses the attention map output by low-level TA-LSTM to help distinguish similar targets in the video, thereby assigning a unique ID to each object with extremely high continuity and real-time speed, realizing the distinction of similar objects in rehabilitation video. Attached Figure Description

[0060] Figure 1 This is a flowchart of a target identity detection system and method for a machine-assisted rehabilitation system according to the present invention;

[0061] Figure 2 This is a target recognition image of a target identity detection system and method for machine-assisted rehabilitation systems according to the present invention;

[0062] Figure 3 This is a structural diagram of the RSSD (Real-Side Detection System) for a target identity detection system and method in a machine-assisted rehabilitation system according to the present invention.

[0063] Figure 4 This is an ST-LSTM model diagram of a target identity detection system and method for machine-assisted rehabilitation systems according to the present invention;

[0064] Figure 5 This is a TA-LSTM model diagram of a target identity detection system and method for machine-assisted rehabilitation systems according to the present invention;

[0065] Figure 6 For the present invention Figure 1 A flowchart of a real-time identity recognition algorithm for a target identity detection system and method used in machine-assisted rehabilitation systems. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0067] Please see Figure 1-6 The present invention provides an embodiment of a target identity detection system for a machine-assisted rehabilitation system, comprising an RSSD module, a TA-LSTM module, a target detection module, and a real-time identity recognition module, characterized in that:

[0068] The RSSD module is used to extract the detection box information of the patient's limbs or assistive tools in each frame of the rehabilitation video;

[0069] The TA-LSTM module detects targets, suppresses useless features to reduce computation, and enhances target features.

[0070] The target detection module detects the bounding box information of the patient's limbs or assistive tools in each frame of the rehabilitation video. The bounding box information includes the target category score, the coordinates of the center point of the bounding box, and its width and height.

[0071] The real-time identity recognition module identifies the target in the dynamic frame and assigns a corresponding identity number.

[0072] In this implementation, such as Figure 2 As shown, the RSSD module replaces the VGG-16 backbone network of the SSD module with a ResNet-101 network. The replaced RSSD module has the following advantages over the SSD network: reduced computational cost and time for extracting representative features from rehabilitation videos; effective combination of low-level to high-level feature maps; improved detection accuracy and speed for small targets in rehabilitation videos; and a three-level feature extraction structure based on the feature pyramid concept to integrate shallow and deep features for subsequent modeling. The benefits of replacing the SSD network's VGG16 backbone network with ResNet-101 include: ResNet-101 has stronger feature extraction capabilities than VGG-16, especially for small target detection; both have similar structural complexity to ensure real-time operation; and the residual structure in ResNet-101 balances the relationship between increased network depth and decreased performance. The principle behind the three-level feature extraction structure designed using the feature pyramid concept to integrate shallow and deep features is that as features are extracted layer by layer, semantic information gradually becomes richer while detailed information is gradually lost. Low-level feature maps rich in detailed information are more beneficial for small target detection, while deep feature maps rich in semantic information are more beneficial for conventional target detection.

[0073] In this implementation, such as Figure 3 As shown, in the RSSD module architecture, the backbone network of RSSD has 1024 channels in Conv4 of the RSSD backbone network and 512 channels in Conv8 of the last four layers of RSSD. According to the idea of ​​feature pyramid, Conv4 and all subsequent convolutional layers are divided into three parts: low-level feature map part, intermediate-level feature map part, and deep-level feature map part. The reason for the change in the number of channels is: by constructing the feature pyramid structure to obtain three levels of feature maps, the subsequent object detection network can balance the detailed information and semantic information; since the feature maps of different layers need to maintain the same number of channels when input into the TA-LSTM network.

[0074] In this implementation, such as Figure 4-5As shown, the TA-LSTM module introduces the spatiotemporal sequence model ST-LSTM to achieve inter-frame mining. Combined with the forget gate and correlation gate in the attention mechanism, a temporal attention mechanism, the long-short time TA-LSTM module, is obtained. The attention mechanism provides the spatiotemporal sequence model with accurate object location regions to selectively update the memory. Simultaneously, the spatiotemporal sequence model provides rich temporal information for the attention process to generate a more accurate attention map. This accurately suppresses useless features and reduces computational cost, achieving enhancement of target features. The useless features are redundant scale features and background features. Since the TA-LSTM module transmits temporal information frame by frame through the attention map, and corresponds to RSSD... The model employs a three-level feature map, learned by three parallel TA-LSTMs, forming an attention cluster network. The low- and mid-level TA-LSTMs focus on location regression, while the high-level TA-LSTM focuses on object classification. This model maintains the horizontal temporal flow while increasing the flow of information from the highest layer to the lowest layer of the next frame, thus better utilizing the temporal context of adjacent frames to improve detection performance and achieve inter-frame detection. This module implements the mutual promotion of the temporal attention mechanism and the TA-LSTM group. The former provides the latter with accurate object location regions through the attention map, while the latter provides rich temporal information for the attention process. The TA-LSTM module can more accurately suppress useless features (redundant scale features and background features), thereby promoting feature enhancement while reducing computational cost. The ST-LSTM module was redesigned in conjunction with the attention mechanism, referencing the low-level TA-LSTM, with the following specific modifications:

[0075] In TA-LSTM, the memory states from t-τ to t-1 are input simultaneously, denoted as... .

[0076] In TA-LSTM, the low-level feature map from RSSD is transformed into a one-dimensional vector, denoted as s, through three convolutional layers (3×3 × 1024, 3×3 × 1024, and 3×3 × 1). t .

[0077] In TA-LSTM, the input gate function i of ST-LSTM is... t and candidate value vector gt The Hadamard product is replaced by a convolution operation to determine the input features that are highly relevant to the patient's hand; the specific calculation formula is as follows:

[0078]

[0079]

[0080] In the formula, It's a convolution operation. It is the hidden state of the previous frame, w is the weight, and b is the bias;

[0081] In TA-LSTM, the forgetting gate function f of ST-LSTM is redesigned by incorporating an attention mechanism. t This is done to retain memory states that are strongly related to the patient's hand and remove redundant memory states; the specific calculation formula is as follows:

[0082]

[0083]

[0084]

[0085] In equation (3), It is an update function, such as E3D-LSTM and self-attention mechanism, but the Hadamard product is replaced by convolution operation; in equation (4), by and Dot product measurement Memory state c of different frames k The similarity between them is expressed by the softmax function output c. k We calculate the weights of each weight and then perform a weighted sum. In equation (5), ⊙ is the Hadamard product, and LayerNorm is the layer normalization used for gradient instability problems.

[0086] In this implementation, the target detection module uses the SSD's Default Box mechanism to obtain the detection box information of the patient's limbs or assistive tools in each frame of the rehabilitation video. The detection box information includes the target category score, the coordinates of the detection box center point, and the width and height. The module locates the position of the detected object in each frame by regressing the coordinates of the detection box center point and the width and height. At the same time, the module classifies the detected object by generating the category score of the detected object and gives the classification confidence score.

[0087] Using the Default Box mechanism in SSD, prediction is performed using three feature maps of different sizes generated by the TA-LSTM group. A series of concentric Default Boxes are generated, centered on the midpoint of each feature on the feature map. The requirements for the Default Box generated for each feature from different TA-LSTM feature maps are shown in the figure below:

[0088] Default Box requires a chart quantity 4 6 4 scale 0.2 0.55 0.9 Aspect Ratio (1，2，0.5) (1，2，0.5，3，1 / 3) (1，2，0.5)

[0089] Calculate the width of the Default Box using the following formula, taking into account both the dimensions and aspect ratio. and height

[0090]

[0091]

[0092] In addition, a Default Box with an aspect ratio of 1 was defined, and the scale was calculated using the following formula:

[0093]

[0094] Then, for each of the c categories (including the background category), c+4 3×3 convolutional kernels are used to predict c target category scores and four bounding box regression parameters (center point coordinates and width and height) for each DefaultBox. Based on the prediction results, positive and negative samples are selected from all DefaultBoxes. Positive samples are first selected according to the following criteria: 1) DefaultBoxes with the highest IoU value matched to any ground truth box; 2) DefaultBoxes with an IoU value greater than 0.5 with any ground truth box. Negative samples are then selected according to the following criteria: For all remaining DefaultBoxes, the maximum confidence score is calculated, and the top-ranked DefaultBoxes are selected as negative samples, ensuring that the number of negative samples is three times the number of positive samples. Finally, the selected DefaultBoxes are fed into the real-time identity recognition module to obtain the final detection results of the patient's limbs or assistive devices in each frame of the rehabilitation video.

[0095] A target detection method for a mechanically assisted rehabilitation system is characterized by: detecting and assigning unique IDs to patient limbs and assistive tools in rehabilitation videos; applying the trained model to the rehabilitation system using a joint training scheme; and the specific implementation steps are as follows:

[0096] Step 1: Use the RSSD module to extract features from static frames in the rehabilitation video;

[0097] Step 2: Use the TA-LSTM module to perform feature enhancement;

[0098] Step 3: Use object detection algorithms to generate candidate bounding boxes for patient limbs or assistive devices in the rehabilitation video;

[0099] Step 4: Use a real-time identity recognition module to perform real-time identity recognition of the patient's limbs or assistive devices in the rehabilitation video.

[0100] In this implementation, the TA-LSTM method for real-time differentiation uses attention maps to represent the differences between different objects. It extracts information from shallow features and reduces the extraction vector of shallow features to lower computational costs. The important parameters involved are as follows: Tubelet and object are represented as tube and obj, respectively.

[0101] The attention feature map obtained through low-level attention clustering is used to calculate the correlation between different hands / sticks in the same frame, denoted as attention correlation AC. i，j The specific calculation formula is as follows:

[0102]

[0103] In the formula, AMV i With AMV j These are the attention feature maps of object i and object j obtained from the low-level attention cluster, respectively.

[0104] For a given object, its adjacent frames f n with f n + 1 The degree of correlation between them is denoted as the predicted neighboring frame correlation degree C. tube(n，n+1） The specific calculation formula is as follows:

[0105]

[0106] In the formula, tube[n] is the frame f n The bounding box for the patient's hand target; the IoU function is used to calculate the correlation of tubes in adjacent frames; tube_len is the frame f n The number of objects detected; n∈{0,1,2,…,N}, where 0 is the nearest object and N is the farthest object; obj is the ID of a specific object.

[0107] In this implementation, the real-time identity recognition module uses online Tubelet analysis to achieve real-time identity recognition of multiple objects. The input of detection box information in each frame is combined with the attention map output by the low-level TA-LSTM to help distinguish similar targets in the video. Then, the non-maximum suppression idea is used to suppress redundant detection boxes and extract the final detection box. Finally, according to the algorithm, a unique and continuous corresponding identity number is assigned to the patient's limbs or assistive tools in the rehabilitation video.

[0108] In this implementation, such as Figure 5As shown, for ease of understanding, the details of the algorithm are explained first. The meanings of the parameters involved in the algorithm that are not mentioned at present are as follows: 1) CAT refers to the category: hand, stick; 2) UNI_ID refers to the identification code of the target: patient A left hand / right hand, patient B left hand / right hand, stick 1 / 2 / 3; 3) tube[CAT] refers to the set of detection boxes corresponding to this category; 4) CAND refers to the candidate box of the target; 5) CAND[ID] refers to the identification number of the candidate box; 6) LoE refers to the length of the element; 7) tMT refers to the tubelet forward threshold; 8) tS refers to the tubelet confidence score; 9) LPs refers to the position parameter of the detection box; Other algorithm descriptions are as follows: 1) Each tubelet includes two key parameters, UNI_ID and CAT; 2) Each object includes three parameters, CS, LPs and AMC; 3) Three key parameters, including tMT, tS and tube_LEN, were selected through separate experiments; 4) obj ( ） It represents the identity of the object, while obj (-1） This indicates that there is no corresponding identity; 6) When the UNI_ID of the object is determined, the attention map output by the tube[CAT] low-level TA-L STM and the detection box output by the detection network can be updated to form the improved TubeIet:

[0109] S1. Traverse the detected categories in the video: hand, stick;

[0110] S1.1 Determine if there is a detection box for this category (e.g., hand). If yes, proceed to S1.2; otherwise, proceed to S1.4.

[0111] S1.2, Traverse the objects of this category with detection boxes (e.g., objects of the hand category: patient A's left / right hand, patient B's left / right hand).

[0112] S1.2.1, The maximum correlation between the object (e.g., patient A's left hand) and the detection frame. The initial value is set to 0;

[0113] S1.2.2, Traverse all bounding boxes that detect this category (e.g., hand);

[0114] S1.2.3 Determine the correlation C between the detection box and the object (e.g., patient A's left hand). obj，tube Is it greater than If yes, proceed to S1.2.4; otherwise, return to S1.2.2 and continue iterating.

[0115] S1.2.4, C obj，tube Assign to At the same time, the value of the detection box is assigned to the candidate box CAND of the object (such as patient A's left hand);

[0116] S1.2.5, Judgment If the threshold tMT is greater than the threshold, proceed to S1.2.6; otherwise, return to S1.2 to continue the iteration.

[0117] S1.2.6, determine the candidate identity obj of the object (e.g., patient A's left hand). CAND[ID] Assign the value to the object and return to S1.2 to continue the traversal. If the traversal ends, proceed to S1.3.

[0118] S1.3, Traverse all bounding boxes that detect this category (e.g., hand);

[0119] S1.3.1 Determine whether the detection box matches an object. If it does, proceed to S1.3.2; otherwise, return to S1.3 to continue iterating.

[0120] S1.3.2 Obtain the maximum correlation C of this category (e.g., hand), and return to S1.3 to continue traversing. If the traversal ends, proceed to S1.4.

[0121] S1.4. Traverse objects that do not match the identity (such as patient B's right hand).

[0122] S1.4.1 Determine whether the confidence level of the candidate number of the object (such as the right hand of patient B) is greater than the confidence level of the detection box. If so, proceed to S1.4.2; otherwise, return to S1.4 to continue traversing.

[0123] S1.4.2, select candidate identity obj CAND[ID] Assign the value to the object (e.g., the right hand of patient B), and return to S1.4 to continue the traversal. If the traversal ends, proceed to S1.5.

[0124] S1.5 Update the detection box for this category (e.g., hand) and return to S1 to continue traversing. If the traversal ends, proceed to S2.

[0125] S2, Output the identity of the object.

[0126] In this embodiment, the joint training scheme for static frames and video streams is as follows: 1) Confidence loss L c ;2) Position loss L l 3) Attention deficit at ;4) Related loss L as Since the training process only takes place in RSSD and TA-LSTM, the identity-aware algorithm does not participate in the training, and the overall training calculation formula is:

[0127] L=αL c, l +δL at +σL as (11)

[0128] In the formula, α, δ, and σ are the weights of each loss;

[0129] L c Training is performed using the Softmax loss function, L l Training is performed using a smoothed L1 loss function, and the joint loss of the two is L. c，l The calculation formula is:

[0130]

[0131] In the formula, N is the number of matched truth boxes, c represents the confidence score, d represents the default box, and g represents the truth box; x i，j Indicates whether the default box and the actual box match (x i，j =0 indicates no match, x i，j =1 indicates a match); β is used to adjust L c and L l The ratio, initially set to 1;

[0132] The attention loss functions (two low-level attention losses, two mid-level attention losses, and two high-level attention losses) are trained using the cross-entropy function. The multi-scale attention losses are then summed to obtain the final loss L. at The calculation formula is:

[0133]

[0134] In the formula, AM g Represents the true attention map. This represents the k-th level attention map for prediction;

[0135] Since some static score calculations can lead to significant biases in the overall sequence during training, and improper correlations between individual local frames and the global sequence can also affect detection results, this invention proposes a self-supervised training method to mitigate pixel-level variations in video detection tasks and maintain temporal consistency across multiple frames. The key idea is to create a list of locally correlated frame scores (SL) to calculate the correlation loss function L. as The calculation formula is:

[0136]

[0137] In the formula, SL t A list of fractions representing time t. S_LEN represents the average score from time 1 to time t-1, and S_LEN represents the length of the selected sequence.

[0138] Working Principle: This invention proposes a deep model with an attention mechanism based on real-time SSD, namely TARSSD. This model is a video-based target identity detection model applied to rehabilitation systems, solving the problem of real-time multi-size target detection and identity recognition with high accuracy and low latency. This invention is applicable to rehabilitation training detection for patients with various physical disabilities. To illustrate the technical solution of this patent in a clear and concise manner, the following explanation will use the training of two stroke patients holding three wooden sticks as an example.

[0139] To address the challenges of target detection and identification in videos, this model can detect multiple targets of varying sizes over time and assign a unique ID to each hand and stick target in real time. Using a joint training scheme, the trained model is applied to a rehabilitation system, and the specific implementation steps are as follows:

[0140] 1. Extract representative spatial domain features from each static frame captured by the camera of the rehabilitation system;

[0141] 2. Based on the spatial domain features extracted from static frames, the temporal domain features between dynamic frames are modeled to achieve feature enhancement for each frame;

[0142] 3. Based on the enhanced feature map of each frame, generate target detections for the hands of the two patients and the three wooden sticks;

[0143] 4. Based on the information obtained from patient target detection, label the left and right hands of the two patients and each stick for identification purposes.

[0144] For those skilled in the art, the present invention is not limited to the details of the exemplary embodiments described above, and the invention can be implemented in other specific forms without departing from the spirit or scope thereof. Therefore, the embodiments of the present invention are exemplary and not restrictive. The scope of the invention is defined by the appended claims rather than the foregoing description, and therefore all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0145] In the description of this invention, unless otherwise stated, the term "multiple" means two or more terms such as "upper," "lower," "left," "right," "inner," "outer," "front end," "rear end," "head," and "tail," indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0146] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

Claims

1. A target identity detection system for a machine-assisted rehabilitation system, comprising an RSSD module, a TA-LSTM module, a target detection module, and a real-time identity recognition module, characterized in that: The RSSD module is used to extract bounding box information of patient limbs or assistive tools in each frame of rehabilitation video. The RSSD module replaces the VGG-16 backbone network of the SSD module with a Resnet-101 network, and combines the idea of ​​feature pyramid to obtain a three-level feature extraction structure to integrate shallow and deep features for subsequent modeling. In the RSSD module architecture, the Conv4 channel in the RSSD backbone network has 1024 channels; the Conv8 channel in the last four layers of RSSD has 512 channels. According to the idea of ​​feature pyramid, Conv4 and all subsequent convolutional layers are divided into three parts: low-level feature map part, intermediate-level feature map part, and deep-level feature map part. The TA-LSTM module receives the three-level features output by the RSSD module, suppresses redundant scale features and useless background features to reduce computational load, and enhances the target features. The TA-LSTM module introduces the spatiotemporal sequence model ST-LSTM to achieve inter-frame mining. Combined with the forget gate and correlation gate in the attention mechanism, a long-short time TA-LSTM module with a temporal attention mechanism is obtained. The attention mechanism provides the spatiotemporal sequence model with accurate object location regions to selectively update the memory. At the same time, the spatiotemporal sequence model provides rich temporal information for the attention process to generate a more accurate attention map. This accurately suppresses useless features and reduces the amount of computation, thereby enhancing the target features, where useless features are redundant scale features and background features. The target detection module receives the enhanced target features output by the TA-LSTM module; The target detection module uses the SSD's Default Box mechanism to obtain the detection box information of the patient's limbs or assistive tools in each frame of the rehabilitation video. The detection box information includes the target category score, the coordinates of the detection box center point, and its width and height. The real-time identity recognition module receives the detection box information output by the target detection module, performs identity recognition on the target in the dynamic frame, and assigns a corresponding identity number.

2. The target identity detection method for a rehabilitation system according to claim 1, characterized in that: The patient's limbs and assistive devices in the rehabilitation video are detected and assigned unique IDs. After applying the trained model to the rehabilitation system using a joint training scheme, the specific implementation steps are as follows: Step 1: Use the RSSD module to extract features from static frames in the rehabilitation video; Step 2: Use the TA-LSTM module to implement feature enhancement; the TA-LSTM method solves the real-time differentiation problem by representing the differences between different objects according to the diagram, extracting information from shallow features and reducing the extraction vector of shallow features to reduce computational cost; Step 3: Use object detection algorithms to generate candidate bounding boxes for patient limbs or assistive devices in the rehabilitation video; Step 4: Use a real-time identity recognition module to perform real-time identity recognition on patient limbs or assistive devices in the rehabilitation video; the method of the real-time identity recognition module is analyzed by an online Tubelet to achieve real-time identity recognition of multiple objects. The input of detection box information in each frame is combined with the input of the attention map output by a low-level TA-LSTM to help distinguish similar targets in the video. Then, the non-maximum suppression idea is combined to suppress redundant detection boxes and extract the final detection box. Finally, according to the algorithm, a unique and continuous corresponding identity number is assigned to the patient limbs or assistive devices in the rehabilitation video.

3. The target identity detection method for a rehabilitation system according to claim 2, characterized in that: The formation of the Tubelet from the attention map output by the low-level TA-LSTM and the detection box output by the detection network involves the following steps: S1, Traverse the categories detected in the video: S1.1 Determine if there is a detection box for this category. If yes, proceed to S1.2; otherwise, proceed to S1.

4. S1.2, Traverse the objects of this category that have detection boxes; S1.2.1, Maximum correlation between the object and the detection box. The initial value is set to 0; S1.2.2, Traverse all detection boxes that detect this category; S1.2.3 Determine the correlation C between the detection box and the object. obj，tube Is it greater than If yes, proceed to S1.2.4; otherwise, return to S1.2.2 and continue iterating. S1.2.4, C obj，tube Assign to At the same time, the value of the detection box is assigned to the candidate box CAND of the object; S1.2.5, Judgment If the threshold tMT is greater than the threshold, proceed to S1.2.6; otherwise, return to S1.2 to continue iterating. S1.2.6, assign the candidate identity obj to this object. CAND[ID] Assign the value to the object and return to S1.2 to continue the traversal. If the traversal ends, proceed to S1.

3. S1.3, Traverse all detection boxes that detect this category; S1.3.1 Determine whether the detection box matches an object. If it does, proceed to S1.3.2; otherwise, return to S1.3 to continue iterating. S1.3.2 Obtain the maximum correlation C of this category, and return to S1.3 to continue traversing. If the traversal ends, proceed to S1.

4. S1.4, Traverse objects that do not match the identity; S1.4.1 Determine if the confidence level of the candidate number of the object is greater than the confidence level of the detection box. If so, proceed to S1.4.2; otherwise, return to S1.4 to continue traversing. S1.4.2, select candidate identity obj CAND[ID] Assign the value to the object and return to S1.4 to continue the traversal. If the traversal ends, proceed to S1.

5. S1.5 Update the detection box for this category and return to S1 to continue the traversal. If the traversal ends, proceed to S2. S2, Output the identity of the object.

4. The target identity detection method for a rehabilitation system according to claim 2, characterized in that: The joint training method of static frames and videos involves the TA-LSTM model generating similar global memories for each frame. When a static frame with occlusion of detection box information or low image quality is detected, the TA-LSTM model can recall the features of previous frames to assist in the detection of the current frame. Training methods include: 1) Confidence loss L c ; 2) Position loss L l 3) Attention deficit at ;4) Related loss L as This training process is performed only in RSSD and TA-LSTM; the identity-aware algorithm is not involved in the training. The overall training calculation formula is as follows: L=αL c,l +δL at +σL as (1) In the formula, α, δ, and σ are the weights of each loss; Lc is trained using the Softmax loss function. l Training is performed using a smoothed L1 loss function, and the joint loss of the two is L. c The formula for calculating l is: In the formula, N is the number of matched truth boxes, c represents the confidence score, d represents the default box, and g represents the truth box; x i，j Indicates whether the default box and the actual box match (x i，j =0 indicates no match, x i，j =1 indicates a match); β is used to adjust L c and L l The ratio, initially set to 1; Each attention loss is trained using the cross-entropy function, and the multi-scale attention losses are summed to obtain the final loss L. at The calculation formula is: In the formula, AM g Represents the true attention map. This represents the k-th level attention map for prediction; The joint training method also employs a self-supervised training strategy to maintain temporal consistency across multiple frames. Specifically, it creates a list of locally associated frame scores SL to calculate the association loss function L. as The calculation formula is: In the formula, SL t A list of fractions representing time t. S_LEN represents the average score from time 1 to time t-1, and S_LEN represents the length of the selected sequence.

Images