Small sample model optimization system and method based on feature decoupling

Abstract

The application relates to the technical field of action recognition in computer vision, and discloses a small sample model optimization system and method based on feature decoupling, wherein the small sample model optimization method based on feature decoupling comprises the following steps: generating sample features for a small sample sign language video sequence; converting semantic frame scores, transition frame scores and redundant frame scores into semantic gating, transition gating and redundant gating respectively; constructing a purified category prototype by using semantic gating weighting, constructing a transition frame boundary constraint loss by using transition gating, and calculating a category discrimination loss based on redundant gating; updating a continuous sign language recognition model according to the total loss, and iteratively updating the purified category prototype; and outputting a recognition result of the small sample sign language video sequence; by using the semantic gating, the transition gating and the redundant gating, different types of frames can play different roles in subsequent optimization, so that the effective utilization rate of training information under a small sample condition is improved.

Description

Technical Field

[0001] This application relates to the field of action recognition technology in computer vision, and more specifically, to a few-shot model optimization system and method based on feature decoupling. Background Technology

[0002] Sign language is a form of language that uses visual information such as hand shapes, hand movements, upper limb postures, and body gestures to express semantics. With the development of computer vision and deep learning technologies, video-based continuous sign language recognition has been applied to scenarios such as barrier-free communication, sign language teaching, intelligent service windows, and video translation. Continuous sign language recognition typically requires extracting hand regions, posture changes, and temporal context from consecutive frames and mapping them to corresponding sign language words or semantic sequences.

[0003] Existing methods often employ convolutional neural networks, recurrent neural networks, temporal convolutional networks, Transformer networks, or skeleton dot graph networks to model features in sign language videos, and combine these with attention mechanisms, keyframe selection, frame sampling, hand cropping, or redundant frame removal to improve recognition performance. Under conditions of sufficient samples, complete annotation, and a stable acquisition environment, these methods can achieve certain results.

[0004] However, in practical applications, the cost of acquiring and annotating sign language videos is high. Continuous sign language samples often only have word-level, sentence-level, or weak temporal labels, lacking precise frame-level semantic boundary annotations. Especially in scenarios with few samples or incremental optimization, the number of available samples for each sign language word class is limited, while videos commonly contain unstable factors such as inter-word transitions, posture adjustments, pauses, hand occlusion, differences in movement speed, and background disturbances. Because these factors are intertwined with the real semantic actions in time, the model struggles to accurately determine which frames primarily carry category semantics and which frames only reflect action connections or repetitive postures.

[0005] Existing processing methods typically treat the entire video as a unified training sequence, or weaken the influence of certain frames only through sampling and attention weights. This makes it difficult to effectively suppress the interference of non-semantic frames on category discrimination information during feature learning. When transition frames, pause frames, or redundant pose frames are incorrectly learned as category criteria, the category representation formed by a small number of samples is easily contaminated, and the boundaries of adjacent sign language words are also prone to shift, leading to confusion in recognition based on similar hand shapes, similar movement trajectories, or differences in expression among different sign language speakers.

[0006] Therefore, existing optimization techniques for small-sample continuous sign language recognition models still have the following technical problems: under the condition of lacking accurate frame-level semantic boundary annotation and limited training samples, it is difficult to stably distinguish semantic effective information, action transition information and redundant interference information in continuous videos, resulting in impure class feature learning and insufficient temporal boundary constraints, thereby reducing the model's recognition robustness and generalization ability in small-sample scenarios. Summary of the Invention

[0007] The purpose of this application is to improve the problems of unstable category feature learning, insufficient temporal boundary differentiation, and decreased model generalization ability caused by the limited number of training samples, lack of frame-level semantic boundary annotation, and the mixing of semantic frames, transition frames, and redundant frames in the scenario of small-sample continuous sign language recognition, and to provide a small-sample model optimization system and method based on feature decoupling.

[0008] This application provides a few-shot model optimization method based on feature decoupling, including the following steps:

[0009] Receive a small sample of sign language video sequences and generate sample features for the small sample of sign language video sequences based on a pre-trained continuous sign language recognition model;

[0010] Based on the sample features, semantic frame scores, transition frame scores, and redundant frame scores are calculated for each frame of a small sample sign language video sequence, and the semantic frame scores, transition frame scores, and redundant frame scores are converted into semantic gating, transition gating, and redundant gating, respectively.

[0011] A purification category prototype is constructed using semantic gating weighting, a transition frame boundary constraint loss is constructed using transition gating, and a category discrimination loss is calculated based on redundant gating.

[0012] The total loss is obtained by jointly optimizing the category discrimination loss, the transition frame boundary constraint loss and the metric learning loss based on the cleaned category prototype. The continuous sign language recognition model is updated with the total loss, and the cleaned category prototype is iteratively updated to obtain the optimized continuous sign language recognition model and the cleaned category prototype.

[0013] The optimized continuous sign language recognition model and the cleaned-up category prototype output the recognition results of a small sample of sign language video sequences.

[0014] Furthermore, methods for generating sample features include:

[0015] For each frame of a small sample sign language video sequence, the hand detection model and the human pose detection model are called to obtain the detection results. The occlusion ratio is calculated based on the detection results, and structured features are generated based on the detection results and the occlusion ratio.

[0016] A pre-trained continuous sign language recognition model is used to perform forward inference on a small sample of sign language video sequences to generate initial pseudo-labels, blank frame markers, and initial boundary region confidence scores. Initial category prototypes are then constructed from the initial pseudo-labels.

[0017] The frame-level visual feature vector is output by the backbone network of the continuous sign language recognition model, and visual sub-features are extracted based on the frame-level visual feature vector.

[0018] Generate sample features, which include structured features, frame-level visual feature vectors, visual sub-features, initial pseudo-labels, blank frame markers, initial boundary region confidence, and initial category prototypes.

[0019] Furthermore, methods for calculating the occlusion ratio include:

[0020] The detection results are organized into anatomical left-hand instances and anatomical right-hand instances, and the instance corresponding to any anatomical hand is called the target hand instance.

[0021] The first criterion is obtained by calculating the proportion of the number of key points with a key point confidence level less than the preset key point visibility threshold in the detection results of the target hand instance to the total number of key points of the target hand instance.

[0022] The second criterion is calculated by the ratio of the area of ​​the hand detection box in the detection results of the target hand instance to the median area of ​​the hand detection box of the same anatomical hand.

[0023] The third criterion is calculated based on the mean confidence of key points in the overlapping area of ​​the hand detection boxes of the anatomical left hand instance and the anatomical right hand instance;

[0024] The maximum value among the first criterion, the second criterion, and the third criterion is taken as the occlusion ratio.

[0025] Furthermore, methods for obtaining semantic gating, transition gating, and redundancy gating include:

[0026] Calculate semantic frame scores based on sample features;

[0027] Transition frame scores are calculated based on frame-level visual feature vectors, the probability of predicting sign language categories using a continuous sign language recognition model, and the confidence level of the initial boundary region.

[0028] The semantic frame score and the transition frame score are respectively processed by threshold shifting and slope amplification, and then processed by the Sigmoid function to obtain semantic gating and transition gating respectively.

[0029] The neighbor-preserving similarity is calculated based on the cosine similarity of adjacent frames in a small sample sign language video sequence. The redundant frame score is calculated based on the neighbor-preserving similarity, semantic gating, and transition gating.

[0030] The redundant frame scores are processed by threshold shifting and slope amplification, and then processed by the Sigmoid function to obtain the redundancy gate.

[0031] Furthermore, methods for calculating semantic frame scores include:

[0032] The confidence level of key hand regions is calculated based on the hand detection confidence level and occlusion ratio in the detection results.

[0033] Semantic proximity is calculated based on the cosine similarity between frame-level visual feature vectors and the prototype of the previous round of purification categories.

[0034] The unstable motion mutation amount is calculated based on the hand motion features in the structured features. The semantic frame score is obtained by weighting and combining the confidence of the key hand region, semantic proximity and unstable motion mutation amount.

[0035] Furthermore, methods for constructing purification category prototypes include:

[0036] For all candidate frames with pseudo-labels of the same sign language category in the k-th iteration, the frame-level visual feature vector of each frame is multiplied by the semantic gate, summed, and then divided by the cumulative term to obtain the weighted average term of the k-th iteration; where the cumulative term is the sum of the semantic gates of all candidate frames plus a preset constant, k is an integer greater than or equal to one, and the zero-th round purification category prototype is the initial category prototype.

[0037] When the number of candidate frames in the k-th iteration is zero or the sum of the semantic gating of the candidate frames is zero, the prototype of the cleanup category for the sign language category in the k-th iteration is the prototype of the cleanup category in the k-th minus one iteration.

[0038] When the sum of the number of candidate frames and the semantic gating is greater than zero, and the number of candidate frames or the sum of the semantic gating of candidate frames in the sign language category in the k-th iteration is less than the corresponding preset stability lower limit, the purification category prototype of the k-th minus one-th round is linearly fused with the weighted average term of the k-th round to obtain the purification category prototype of the k-th round.

[0039] Otherwise, take the weighted average term of the kth round as the prototype of the purification category of the kth round.

[0040] Furthermore, methods for calculating the boundary constraint loss of the transition frame include:

[0041] Based on the initial pseudo-label, different preceding and following categories are determined for each frame, and frames that simultaneously exist in both preceding and following categories are taken as valid transition candidate frames.

[0042] Within the preset neighborhood window of the effective transition candidate frames, the frame with the largest semantic gating on both the previous and next categories is taken as the reference semantic frame for the corresponding category.

[0043] Based on the effective transition candidate frames, the frame-level visual feature vectors of the reference semantic frames of the previous and next categories, and the current reference cleanup category prototypes of the previous and next categories, the boundary margin penalty terms of the previous and next categories are calculated; the current reference cleanup category prototype is the cleanup category prototype of the k-th round.

[0044] Based on the transition gating of effective transition candidate frames and the boundary interval penalty terms between the previous and next categories, the boundary loss term is calculated. The multiple boundary loss terms are accumulated to obtain the transition frame boundary constraint loss.

[0045] Furthermore, methods for obtaining the total loss include:

[0046] The learning loss is calculated based on semantic gating, frame-level visual feature vectors, and the current reference cleanup category prototype.

[0047] The neighborhood consistency constraint loss is calculated by weighting the difference between the frame-level visual feature vector and the mean of the frame-level visual feature vector of its neighboring frames using redundancy gating.

[0048] A gate coordination constraint loss is constructed based on semantic gating, redundant gating, and transition gating;

[0049] Based on visual sub-features, we calculate the decoupling orthogonal constraint loss, sub-feature reconstruction loss, and semantic sub-feature-assisted category discrimination loss.

[0050] The total loss is calculated based on the category discrimination loss, metric learning loss, transition frame boundary constraint loss, neighborhood consistency constraint loss, decoupling orthogonality constraint loss, sub-feature reconstruction loss, semantic sub-feature-assisted category discrimination loss, and gating coordination constraint loss.

[0051] Furthermore, methods for calculating the decoupling orthogonal constraint loss, sub-feature reconstruction loss, and semantic sub-feature-assisted category discrimination loss include:

[0052] Visual sub-features include semantic sub-features, transition sub-features, and redundant sub-features;

[0053] The decoupling orthogonal constraint loss is obtained by summing the squares of the cosine similarities between each pair of normalized semantic sub-features, normalized transition sub-features, and normalized redundant sub-features.

[0054] The semantic sub-features, transition sub-features, and redundant sub-features are mapped by their respective back-projection matrices and then added together to obtain the frame-level reconstructed feature vector. The sub-feature reconstruction loss is obtained by summing the squared L2 norm of the difference vector between the frame-level reconstructed feature vector and the frame-level visual feature vector.

[0055] Based on the category probabilities and pseudo-labels output by semantic sub-features, calculate the semantic sub-feature-assisted category discrimination loss.

[0056] This application provides a few-shot model optimization system based on feature decoupling, used to execute the aforementioned few-shot model optimization method based on feature decoupling; the system includes:

[0057] The feature representation module is used to receive small sample sign language video sequences and generate sample features for the small sample sign language video sequences based on a pre-trained continuous sign language recognition model.

[0058] The decoupled scoring module calculates semantic frame scores, transition frame scores, and redundant frame scores for each frame of a small sample sign language video sequence based on sample features, and converts the semantic frame scores, transition frame scores, and redundant frame scores into semantic gating, transition gating, and redundant gating, respectively.

[0059] The purification and weight reduction module is used to construct purification category prototypes with semantic gating weighting, construct transition frame boundary constraint loss with transition gating, and calculate category discrimination loss based on redundant gating.

[0060] The loss optimization module is used to jointly optimize the category discrimination loss, the transition frame boundary constraint loss and the metric learning loss based on the cleaned category prototype to obtain the total loss. The total loss is used to update the continuous sign language recognition model, and the cleaned category prototype is iteratively updated to obtain the optimized continuous sign language recognition model and the cleaned category prototype.

[0061] The recognition output module outputs the recognition results of small sample sign language video sequences based on the optimized continuous sign language recognition model and the cleaned category prototype.

[0062] The beneficial effects of this application are as follows: This application generates sample features from small sample sign language video sequences, and calculates semantic frame scores, transition frame scores and redundant frame scores based on the sample features, so that the mixed category semantic information, action switching information and repetitive interference information in continuous sign language videos can be distinguished and represented, avoiding the treatment of various types of frames in the entire video as homogeneous training data.

[0063] This application converts semantic frame scores, transition frame scores, and redundant frame scores into semantic gating, transition gating, and redundant gating, respectively, so that different types of frames play different roles in subsequent optimization. Semantic gating is used to highlight frames that stably carry sign language category information, transition gating is used to constrain the switching region between adjacent sign language words, and redundant gating is used to reduce the impact of repeated poses, pause frames, and non-semantic perturbations on category discrimination, thereby improving the effective utilization rate of training information under small sample conditions.

[0064] This application constructs a cleaned category prototype by using semantic gating weighting, which can reduce the pollution of the category prototype by transition frames and redundant frames, and enable a small number of effective semantic frames to form a more stable category representation. At the same time, constructing transition frame boundary constraint loss with transition gating is beneficial to enhancing the boundary discrimination ability between adjacent sign language words and reducing category confusion caused by similar hand shapes or similar movement trajectories.

[0065] This application jointly optimizes the category discrimination loss, transition frame boundary constraint loss, and metric learning loss based on the cleaned category prototype. During the optimization process, iteratively updates the continuous sign language recognition model and the cleaned category prototype, enabling synergistic effects between model parameter updates, category prototype cleanup, and temporal boundary correction. Therefore, this application can improve the robustness, category discrimination ability, and generalization performance across sign speakers and scenarios of the continuous sign language recognition model, even under conditions of insufficient precise frame-level semantic boundary annotation and a limited number of training samples. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the few-sample model optimization method based on feature decoupling proposed in this application.

[0067] Figure 2 This is an example diagram illustrating the construction of the purification category prototype, the construction of the transition frame boundary constraint loss, and the calculation of the category discrimination loss in this application;

[0068] Figure 3 This is a module example diagram of the few-sample model optimization system based on feature decoupling proposed in this application. Detailed Implementation

[0069] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0070] A few-sample model optimization system and method based on feature decoupling, including the following embodiments:

[0071] Example 1:

[0072] The feature-decoupling-based few-sample model optimization method is initiated during the few-sample training or incremental optimization phase of the continuous sign language recognition model. It can be applied to scenarios such as sign language teaching systems, barrier-free communication terminals, sign language video translation platforms, and sign language recognition devices for intelligent service windows.

[0073] In this embodiment, each type of sign language word or each sign language segment has only a small number of training samples. The video samples have word-level labels, sentence-level labels, or weak temporal labels, but lack precise frame-level semantic boundary annotations. The video contains individual differences among sign language users, differences in movement speed, differences in pauses, differences in transitional movements, hand occlusion, and background perturbations. A hand detection model, a human posture detection model, and a pre-trained continuous sign language recognition model are pre-configured to generate initial frame-level features and initial boundary region confidence scores.

[0074] The pre-trained continuous sign language recognition model consists of a frame-level feature extraction backbone network and a temporal modeling network. The frame-level feature extraction backbone network is a convolutional neural network that maps a single frame image into a frame-level visual feature vector. The temporal modeling network models the frame-level feature sequence and outputs the predicted probability of each frame belonging to each sign language category.

[0075] In an optional embodiment, the temporal modeling network employs a temporal convolutional network, a recurrent neural network, or a Transformer coding network; all three types of networks receive frame-level feature sequences and output a class probability sequence corresponding to the total number of video frames T, with the class set, whitespace setting, and output frame order remaining consistent throughout the training, validation, inference, and online update phases.

[0076] Small sample model optimization methods based on feature decoupling, such as Figure 1 As shown, it includes the following steps:

[0077] Step 100: Receive a small sample sign language video sequence and generate sample features for the small sample sign language video sequence based on a pre-trained continuous sign language recognition model.

[0078] Step 100 is used to receive a small sample sign language video sequence and generate sample features based on the small sample sign language video sequence. The small sample sign language video sequence is composed of the first frame image to the Tth frame image, where T represents the total number of video frames. In this embodiment, small sample means that the number of training video samples corresponding to each sign language category is less than or equal to a preset small sample number threshold. The preset small sample number threshold is determined according to the distribution of the number of samples of each category in the training set. The number of training video samples corresponding to each sign language category in the training set is sorted from smallest to largest, and the lower quartile of the sample number distribution is taken and rounded to a positive integer. The obtained positive integer is used as the preset small sample number threshold. After size normalization and pixel normalization of each frame, the hand detection model and human pose detection model are called for each frame image to obtain the detection results. The detection results include hand detection. The system assigns bounding boxes and their confidence scores, upper limb keypoint coordinates and their confidence scores, and hand keypoint coordinates and their confidence scores. Following the aforementioned hand allocation method, the system organizes the detection results into anatomical left-hand instances and anatomical right-hand instances, and refers to the instance corresponding to any anatomical hand as the target hand instance. If a certain anatomical hand does not have a matching hand detection box in the current frame, but the hand keypoints of that anatomical hand can be output by the human pose detection model or interpolated from adjacent frames, then the bounding rectangle of the hand keypoints of that anatomical hand is used as the hand detection box of the target hand instance, and the average confidence score of the hand keypoints of that anatomical hand is used as the hand detection confidence score of the target hand instance. If neither the hand detection box nor the hand keypoints are available, then the hand detection confidence score of the target hand instance is zero, the hand keypoint confidence score is zero, and the occlusion ratio of the target hand instance is one.

[0079] The small sample sign language video sequences are input in frame-by-frame order preservation because the semantics of continuous sign language are determined by hand shape, movement trajectory, and the switching of adjacent sign language words. If only isolated images are input, the transition position and the duration of the posture will be lost. Size normalization and pixel normalization bring single-frame images from different shooting resolutions and brightness ranges into the same numerical scale, which makes it easier for the hand detection model, human posture detection model, and frame-level feature extraction backbone network to share subsequent thresholds and statistics.

[0080] In this embodiment, the anatomical left hand or anatomical right hand assigned by hand type in the same frame is referred to as the target hand instance. The target hand instance includes the anatomical hand type, hand detection box, hand detection confidence score, coordinates of the hand keypoints corresponding to that anatomical hand type, and the confidence score of the hand keypoints corresponding to that anatomical hand type. Anatomical left hand and anatomical right hand refer to the left and right hands in the anatomical sense of the sign language interpreter, not based on their left or right position in the image. The same anatomical hand type means that the target hand instances being compared, statistically analyzed, or associated all belong to the same side of the sign language interpreter in the anatomical sense, not the left or right side in the image coordinates. The upper limb keypoints output by the human posture detection model include the anatomical left wrist keypoint and the anatomical right wrist keypoint. The assignment of the hand detection box and hand keypoints is determined by the wrist coordinates. Keypoints and hand detection boxes are matched. For each hand detection box, the distance from the center of the detection box to the anatomical left wrist keypoint and the anatomical right wrist keypoint is calculated. Combined with the average confidence score of the hand keypoints of the corresponding anatomical hand type falling into the detection box, the anatomical hand type with the lowest matching cost is selected as the anatomical hand type of the detection box. When the human pose detection model has already output hand keypoints with labels of anatomical left hand or anatomical right hand, the label is directly used, and the matching is completed by the intersection-union ratio of the hand detection box and the bounding region of the hand keypoint under the label. For the same anatomical hand type in adjacent frames, the system performs temporal association based on the hand center position, the intersection-union ratio of the detection box, and the similarity of the hand keypoint coordinates, so that the same target hand instance maintains the same anatomical hand type in the video sequence.

[0081] Specifically, for any hand detection frame, the distances from the center of the hand detection frame to the key points of the anatomical left wrist and the anatomical right wrist are calculated respectively, and the distances are normalized to obtain the positional distance term for the corresponding anatomical hand; the mean confidence score of the key points of the corresponding anatomical hand falling within the hand detection frame is calculated, and the mean confidence score of the key points is subtracted from the mean confidence score to obtain the confidence cost term for the corresponding anatomical hand; the positional distance term is multiplied by the positional weight, and the confidence cost term is multiplied by the preset confidence weight to obtain the matching cost for the corresponding anatomical hand, wherein the sum of the positional weight and the preset confidence weight is one.

[0082] The pre-set confidence weights are determined through the validation set. Specifically, multiple candidate confidence weights are generated within a closed interval of zero to one. For each candidate confidence weight, one is subtracted from the candidate confidence weight to obtain the corresponding position weight. The matching cost is then used to complete the anatomical hand assignment for each hand detection box in the validation set. The anatomical hand assignment accuracy corresponding to each candidate confidence weight is calculated, and the candidate confidence weight that maximizes the anatomical hand assignment accuracy is taken as the pre-set confidence weight. When multiple candidate confidence weights correspond to the same highest anatomical hand assignment accuracy, the median of the multiple candidate confidence weights is taken as the pre-set confidence weight. The pre-set confidence weights remain consistent throughout the training, validation, inference, and online update phases.

[0083] In an optional embodiment, the hand assignment method further includes mirror input processing; when receiving video, the system reads a mirror mark, which indicates whether the input image has been flipped horizontally; when the mirror mark indicates that there is a horizontal flip, the system only performs a reverse transformation on the image coordinates without changing the definition of the anatomical left hand and the anatomical right hand; if the mirror mark is missing, the system determines the mapping from the image coordinates to the anatomical hand based on the relative relationship between the shoulder key points, wrist key points and hand movement trajectories in consecutive frames, and fixes this mapping for the same small sample sign language video sequence.

[0084] The hand detection model is a target detection model based on convolutional neural networks. Its input is a single-frame image, and its output is the bounding box coordinates of the hand region and the corresponding detection confidence score. The human pose detection model is a keypoint regression model based on convolutional neural networks. Its input is a single-frame image, and its output is the coordinates of keypoints of the upper limbs such as the shoulder, elbow, and wrist, as well as keypoints of the hand and their confidence scores.

[0085] Both the hand detection model and the human pose detection model use single-frame images as input because the hand region, upper limb keypoints, and hand keypoints are intra-frame observable structures. Single-frame input can generate the spatial localization information required for the target hand instance without waiting for a complete video. The training steps for both are as follows: using a dataset labeled with hand bounding boxes, upper limb keypoints, and hand keypoints as training samples, the parameters of the hand detection model are updated using bounding box regression loss and detection classification loss, and the parameters of the human pose detection model are updated using keypoint coordinate regression loss and keypoint confidence loss. Then, on the validation set, preset keypoint visibility thresholds, preset area anomaly ratios, and detection confidence correlation thresholds are determined. Finally, the trained model is fixed for feature generation in step 100.

[0086] For each target hand instance, the occlusion ratio is calculated separately. The occlusion ratio is not determined solely by the overlap area between the hand detection box and the upper limb region, as overlap between the hand and forearm / upper limb regions is common in sign language and does not necessarily represent occlusion. The occlusion ratio is derived from three complementary criteria: the first criterion is the visibility of hand keypoints, calculated as the proportion of keypoints with a confidence level less than a preset keypoint visibility threshold in the target hand instance relative to the total number of keypoints in the target hand instance; the second criterion is the abnormality of the hand detection box area, calculated by comparing the area of ​​the hand detection box of the target hand instance in the current frame with that of the same anatomical hand within the current small sample sign language video sequence. The second criterion is determined by comparing the median area of ​​the hand detection bounding box with the median area. If the area is less than the median multiplied by a preset abnormal area ratio, it indicates that the hand is outside the frame or occluded. The ratio is then subtracted from the median area of ​​the hand detection bounding box of the target hand instance in the current frame. The third criterion is the degree of mutual occlusion between the left and right hands. When the hand detection bounding boxes of the dissected left and right hand instances overlap, the average confidence scores of the hand keypoints of the two target hand instances are calculated within the overlapping area. If the two averages are not equal, the target hand instance with the minimum average is identified as the occluded hand instance in the mutual occlusion determination, and the ratio of the overlapping area of ​​the two hand detection bounding boxes to the minimum area of ​​the two hand detection bounding boxes is taken as the occluded hand instance. The third criterion value is set to zero for the other target hand instance; when the two means are equal and the confidence scores of the two hand detections are unequal, the target hand instance with the minimum hand detection confidence score is taken as the occluded hand instance, and the overlap ratio is used as the third criterion value for the occluded hand instance, while the third criterion value for the other target hand instance is set to zero; when the mean confidence score of the hand keypoints and the confidence score of the hand detection are equal, the third criterion value for both target hand instances is the overlap ratio; the occlusion ratio of the target hand instance is the maximum value of the above three criteria, so that the occlusion ratio increases only when occlusion evidence such as invisible keypoints, abnormal detection box area, or mutual occlusion of left and right hands appears, while the overlap of the hand and upper limb regions under normal posture is not judged as occlusion; frame level The occlusion ratio is the maximum of the occlusion ratios of the left-hand and right-hand anatomical examples. The frame-level hand detection confidence is the arithmetic mean of the hand detection confidence of the left-hand and right-hand anatomical examples. The preset keypoint visibility threshold is the lower quartile of the keypoint confidence distribution when the human pose detection model correctly detects keypoints on the validation set. The preset area anomaly ratio is the lower quartile of the distribution of the ratio of hand detection box areas in adjacent frames of the same anatomical hand in the training set. When there is no matched hand detection box area for a certain anatomical hand in the current small sample sign language video sequence, the median of the hand detection box area for that anatomical hand is the median of the distribution of matched hand detection box areas for the same anatomical hand in the training set.

[0087] In an optional embodiment, when the area of ​​the target hand instance's hand detection box is greater than or equal to the median area of ​​the hand detection boxes of the same anatomical hand multiplied by a preset area anomaly ratio, the second criterion is set to zero; when the hand detection boxes of the anatomical left hand instance and the anatomical right hand instance do not overlap, the third criterion for both target hand instances is set to zero.

[0088] In an optional embodiment, when the hand detection box of a target hand instance exceeds the effective area of ​​the image, the system first crops the hand detection box into the effective area of ​​the image, and then calculates the area of ​​the hand detection box, the overlapping area, and the average confidence of the hand key points falling into the detection box; if the area after cropping is smaller than the area before cropping, the target hand instance is considered to have a truncated image boundary; the ratio term of the second criterion is the quotient of the area after cropping divided by the median area of ​​the hand detection boxes of other anatomical hands.

[0089] Based on the aforementioned target hand instances and the pre-trained continuous sign language recognition model, a frame-level feature extraction backbone network is used to obtain the structured features of each frame. Each frame's structured features are composed of concatenated sub-parts. The first part is the hand shape feature, consisting of finger bending angles, palm orientation, fingertip distance, and the relative positions of the left and right hands. The finger bending angle is estimated by the angle between the lines connecting adjacent phalanges in the same target hand instance. When only a monocular two-dimensional image is available, a two-dimensional palm face proxy orientation is used. This proxy orientation is determined by key points in the palm region of the same target hand instance. Specifically, the vector pointing from the wrist key point to the metacarpophalangeal joint key point at the base of the middle finger is used as the palm length direction, and the vector pointing from the metacarpophalangeal joint key point at the base of the index finger to the metacarpophalangeal joint key point at the base of the little finger is used as the palm width direction. The palm length and palm width directions are then combined... The symbol encoding of the cross product direction of the two is used as the palm orientation feature; the fingertip distance is composed of the distance between each fingertip keypoint in the same target hand instance and the distance between the corresponding fingertip keypoints in the anatomical left hand instance and the anatomical right hand instance. The relative position of the left and right hands is obtained by subtracting the center coordinates of the anatomical right hand instance from the center coordinates of the hand of the anatomical left hand instance. The center coordinates of the hand are taken as the arithmetic mean of the coordinates of the corresponding target hand instance's hand keypoints. Considering the uncertainty of finger bending angle and palm orientation in monocular color images due to estimation by two-dimensional keypoints, this step attaches a reliability weight to each component in the hand shape feature, which is obtained by averaging the confidence of the relevant hand keypoints. The component is multiplied by its reliability weight and then included in the structured feature, so that the component with a keypoint confidence of zero does not participate in the contribution of the structured feature.

[0090] In an optional embodiment, when the system is equipped with a depth camera or a 3D hand pose estimator, the palm orientation is obtained from the 3D palm normal. The system selects the wrist key point, the metacarpophalangeal joint key point at the base of the index finger, and the metacarpophalangeal joint key point at the base of the little finger within the same target hand instance to determine the palm surface, and uses the normal of the palm surface as the palm orientation feature. Both the 3D palm normal and the 2D palm proxy orientation are recorded as palm orientation features in the hand shape features, without being named separately.

[0091] The training steps for the pre-trained continuous sign language recognition model are as follows: First, a pre-training dataset is constructed using continuous sign language videos with word-level or sentence-level labels. Size normalization, pixel normalization, and frame order preservation are performed on each video segment. Next, a frame-level feature extraction backbone network extracts frame-level visual feature vectors, and a temporal modeling network outputs the class probability for each frame. Cross-entropy loss is used for training when frame-level labels are available, while CTC loss is used when only word-level or sentence-level labels are available. After training until the recognition accuracy on the validation set stops increasing or a preset number of training rounds is reached, the frames are saved. The parameters of the feature extraction backbone network and the temporal modeling network are included. The validation set recognition accuracy stops increasing when the validation set recognition accuracy does not exceed the previous highest validation set recognition accuracy for a consecutive preset number of rounds. The preset number of rounds is determined by comparing the final recognition accuracy corresponding to different candidate number of rounds on the validation set. This pre-trained continuous sign language recognition model is used to provide transferable frame-level visual feature vectors, initial frame-level pseudo-labels, and initial boundary region confidence in the small sample stage, so that subsequent decoupling optimization does not depend on training from scratch.

[0092] In this embodiment, the key points of the palm region are the set of key points of the hand located in the palmar skeleton of the target hand instance, including wrist key points, key points of the metacarpophalangeal joints at the base of the thumb, key points of the metacarpophalangeal joints at the base of the index finger, key points of the metacarpophalangeal joints at the base of the middle finger, key points of the metacarpophalangeal joints at the base of the ring finger, and key points of the metacarpophalangeal joints at the base of the little finger. If the human posture detection model uses different key point naming methods, a mapping is established according to the key points in the model key point topology corresponding to the above anatomical parts. This mapping remains consistent during the training and inference phases. When a key point of a target hand instance is missing, linear interpolation is first performed based on the temporal correlation results of the same anatomical hand in the preceding and following frames. If the key point cannot be provided in either the preceding or following frame, the confidence of the key point is set to zero, and its contribution is reduced in the feature components calculated from the key point through a reliability weight.

[0093] In an optional embodiment, when the consecutive missing key points of the same anatomical hand reach the upper limit of the preset interpolation span, the system stops using linear interpolation across the missing segment, directly sets the confidence of the missing key point to zero, and processes the hand shape feature component and hand motion feature component calculated by the missing key point according to the reliability weight; the upper limit of the preset interpolation span is taken as the upper quartile of the distribution of consecutive missing frames of correctly detected hand key points in the training set, and the same value is used in the training, verification, inference and online update stages.

[0094] The use of keypoint mapping in the palm region and linear interpolation of the same anatomical hand is to enable pose detection models with different keypoint topologies to output the same structured features and to maintain the continuity of the feature sequence of the same anatomical hand when short-term detection is missing; the processing of zero confidence of keypoints prevents missing information from being mistakenly used as input to determine hand shape.

[0095] The second part consists of hand motion features, comprising the hand center displacement, keypoint velocity, and trajectory curvature of adjacent frames. The hand center displacement and keypoint velocity of adjacent frames are calculated based on the differences in hand center coordinates and keypoint coordinates between adjacent frames for the same anatomical hand instance. The trajectory curvature is calculated based on the angle of the broken line formed by the hand center coordinates of three consecutive frames for the same anatomical hand. For the first and last frames of the video, the hand center displacement, keypoint velocity, and trajectory curvature are taken from the corresponding values ​​of adjacent computable frames. When T is less than three, the trajectory curvature is zero; when T is equal to one, the hand center displacement and keypoint velocity are zero vectors. The third part consists of image quality and occlusion features, comprising frame-level hand detection confidence, frame-level occlusion ratio, and motion blur. The motion blur is calculated as follows: for the image regions bounded by the hand detection boxes of the anatomical left hand instance and the anatomical right hand instance, the pixel gray-level gradient magnitude is calculated along the horizontal and vertical directions respectively. The variance of the gradient magnitude of all pixels within each hand detection box is taken as the sharpness measure of the target hand instance. The arithmetic mean of the sharpness measures of the two target hand instances is taken as the sharpness measure of the hand region of the frame. The motion blur is obtained by subtracting the value of the sharpness measure after normalization function from the value of the normalization function. The motion blur increases when the gradient variance decreases and decreases when the gradient variance increases. The statistical minimum and statistical maximum values ​​of the normalization function are taken from the global distribution of the gradient variance of the hand region on the training set.

[0096] The structured features select hand shape features, hand movement features, and image quality and occlusion features as inputs because sign language categories are jointly determined by hand shape, movement direction, relative position of upper limbs, and observable quality. Using these inputs in parallel with frame-level visual feature vectors can provide interpretable supplementary information for the construction of scoring, gating, and purification category prototypes when the number of visual appearance samples is limited.

[0097] The normalization function uses min-max normalization, mapping a set of values ​​to a closed interval between zero and one. Its calculation method is as follows: for a given value of a variable, subtract the variable's statistical minimum value from that value, then divide by a normalization term. The normalization term is the difference between the variable's statistical maximum and minimum values ​​plus a preset constant. In this embodiment, all preset constants used for the denominator are for numerical stability, configured as positive numbers, and determined according to one-hundredth of the first percentile of the non-zero values ​​of the corresponding denominator in the training set. If the corresponding denominator has no non-zero values ​​in the training set, the preset constant is set to one millionth. The preset constant remains consistent throughout the training, validation, inference, and online update phases, does not change with individual video samples, and does not participate in model learning. This setting method is used for all subsequent occurrences of the preset constant. For variables with consistent physical meaning across different videos, such as hand detection confidence, hand key region confidence, and unstable motion mutation, their statistical minimum and maximum values ​​are taken from the training... The global distribution of the variable on the training set ensures that the same physical value is mapped to the same normalized result across different videos, maintaining cross-video comparability. For variables compared only within a single video sequence, their statistical minimum and maximum values ​​are taken from the current video sequence. When the difference between the statistical maximum and minimum values ​​of a variable is less than a preset normalization degradation threshold, i.e., when the variable's value is approximately constant within the statistical range, the normalization function outputs a neutral value of 0.5 for all values ​​of the variable, instead of all zeros, to avoid variables with approximately constant values ​​being incorrectly mapped to zero, thus weakening subsequent scores. When the difference is not less than the preset normalization degradation threshold, the output corresponding to the statistical minimum is zero, the output corresponding to the statistical maximum is one, and the remaining values ​​are mapped linearly to the range between zero and one. The preset normalization degradation threshold is one-tenth of the standard deviation of the variable's values ​​on the training set; that is, when the difference between the statistical maximum and minimum values ​​of the variable is less than one-tenth of the variable's standard deviation, degradation output is triggered.

[0098] Simultaneously, the frame-level feature extraction backbone network outputs a frame-level visual feature vector for each frame. Based on the frame-level visual feature vector, visual sub-features are extracted, including semantic sub-features, transition sub-features, and redundant sub-features. Specifically, three parallel linear projection heads are set on top of the frame-level visual feature vector, referred to as the semantic projection head, transition projection head, and redundant projection head, respectively. The semantic projection head multiplies the frame-level visual feature vector by a semantic projection matrix to obtain semantic sub-features; the transition projection head multiplies the frame-level visual feature vector by a transition projection matrix to obtain transition sub-features; and the redundant projection head multiplies the frame-level visual feature vector by a redundant projection matrix to obtain redundant sub-features. The semantic sub-features, transition sub-features, and redundant sub-features directly output by the projection heads retain amplitude information and are used for sub-feature reconstruction loss. When participating in the decoupling orthogonal constraint loss, the system further performs L2 normalization on the three sub-features to obtain normalized semantic sub-features, normalized transition sub-features, and normalized redundant sub-features.

[0099] The L2 norm normalization is calculated by dividing each sub-feature by a norm term, where the norm term is the L2 norm of the sub-feature itself. This makes each normalized sub-feature a unit-length vector, thus preventing sub-features from collapsing into zero vectors and eliminating the numerical instability of cosine similarity near zero vectors. When a sub-feature is a zero vector, the norm term is the L2 norm of the sub-feature itself plus a preset constant. To prevent the three projection heads from simultaneously compressing the frame-level visual features into meaningless directions, the three sub-features must also satisfy a reconstruction constraint. That is, the three sub-features that have not undergone L2 norm normalization are mapped back to the frame-level visual feature space through their respective back-projection matrices and then added together to form a frame-level reconstructed feature vector, ensuring that the frame-level reconstructed feature vector is consistent with the original frame-level visual feature vector. Maintaining close proximity, this reconstruction constraint serves as a component of subsequent decoupling optimization, ensuring that the three sub-features as a whole retain the information of the frame-level visual features without degradation. The frame-level visual feature vector is used for subsequent score calculation, initial category prototype construction, cleansing category prototype construction, and metric learning. The normalized semantic sub-features, normalized transition sub-features, and normalized redundant sub-features are used for the decoupling orthogonal constraint in step 400. The semantic sub-features are also connected to category discrimination supervision, that is, a semantic sub-feature auxiliary classifier that shares the category set with the temporal modeling network is set on top of the semantic sub-features, so that the semantic sub-features maintain category discrimination ability and avoid minimizing the cosine similarity between sub-features alone, which would cause the projection head to output a direction without discriminative meaning.

[0100] For example, the first The frame-level visual feature vector of a frame is denoted as The three categories of meaning—semantic, transitional, and redundant—are represented by superscripts. , and The semantic projection matrix, transition projection matrix, and redundant projection matrix are denoted as follows: , and Then, the semantic sub-features, transition sub-features, and redundant sub-features can be represented as: , , ;in, Indicates the first Semantic sub-features of a frame Indicates the first Frame transition features Indicates the first Redundancy sub-features of frames Indicates the frame number.

[0101] For example, normalized semantic sub-features, normalized transition sub-features, and normalized redundant sub-features can be represented as follows: , , ,when , and When it is a zero vector, , , ;in, , and They represent the first The frame's normalized semantic sub-features, normalized transition sub-features, and normalized redundancy sub-features. Represents the L2 norm, Input placeholders representing functions or norms. The term represents a predefined constant used for normalizing the denominator term using the sub-feature 2-norm; the semantic backprojection matrix, transition backprojection matrix, and redundant backprojection matrix are denoted as follows: , and Then the first The frame-level reconstruction feature vector of a frame can be represented as: ,in Indicates the first Frame-level reconstructed feature vectors of a frame.

[0102] Cosine similarity is used to measure the degree of closeness between two vectors in terms of direction. It is calculated by dividing the inner product of the two feature vectors by the product of their respective L2 norms. The value of cosine similarity is a closed interval from negative one to positive one. It takes a value of positive one when the two vectors are in the same direction, a value of negative one when the two vectors are in opposite directions, and a value of zero when the two vectors are orthogonal.

[0103] The aforementioned backprojection matrices include a semantic backprojection matrix, a transitional backprojection matrix, and a redundant backprojection matrix, each corresponding one-to-one with a semantic sub-feature, a transitional sub-feature, and a redundant sub-feature, respectively. The backprojection matrix is ​​a learnable linear transformation that maps the corresponding sub-feature from the sub-feature space back to the frame-level visual feature space. Its input dimension is consistent with the corresponding sub-feature dimension, and its output dimension is consistent with the frame-level visual feature vector dimension. The backprojection matrix is ​​neither a fixed transpose of the corresponding projection matrix nor a fixed inverse of the corresponding projection matrix. During initialization, the backprojection matrix is ​​initialized using linear layer parameters that match its input and output dimensions. During training, the backprojection matrix parameters, along with the semantic projection matrix, the transitional projection matrix, and the redundant projection matrix, participate in the backpropagation update of the total loss, and the sub-feature reconstruction loss constrains its learning of the reconstruction mapping from the sub-feature space to the frame-level visual feature space.

[0104] In an optional embodiment, the initialization of the backprojection matrix can use the transpose of the corresponding projection matrix as an initialization reference; this initialization reference is only used for given initial parameters and does not imply that the semantic backprojection matrix, the transition backprojection matrix, and the redundant backprojection matrix are respectively restricted to the transpose of the corresponding projection matrix during training.

[0105] In this embodiment, feature decoupling is accomplished by sub-feature projection and gated splitting. Sub-feature projection refers to inputting the same frame-level visual feature vector into the semantic projection head, transition projection head, and redundant projection head respectively to obtain three learnable projection results. The semantic sub-feature is used to carry the direction related to sign language category discrimination, the transition sub-feature is used to carry the direction related to the switching of adjacent segments, and the redundant sub-feature is used to carry the direction related to pauses, repetition maintenance, and non-semantic perturbations. The three originate from the same frame-level visual feature vector, but in step 400, they are assigned to different directions by the decoupling orthogonal constraint that the square of the pairwise cosine similarity of the normalized sub-features tends to zero. After the three are synthesized by the sub-feature reconstruction loss constraint, the information of the original frame-level visual feature vector is still retained. Therefore, the decoupling process is not to delete feature components, but to allocate the visual information of the same frame to the three learnable subspaces of semantic, transition, and redundancy.

[0106] Gated decomposition refers to the process in step 200 where semantic frame scores, transition frame scores, and redundant frame scores are converted into semantic gating, transition gating, and redundant gating, respectively, through a preset gating threshold, a preset gating slope coefficient, and a sigmoid function. Semantic gating determines the contribution of the frame to the cleaned category prototype, metric learning, and semantic sub-feature-assisted category discrimination; transition gating determines the contribution of the frame to the transition frame boundary constraints and sign language word switching candidate boundaries; and redundant gating determines the contribution of the frame to the weighting coefficient in the category discrimination loss and the neighborhood consistency constraint. The three gatings and three sub-features correspond to the semantic, transition, and redundant components of the same frame. In each iteration, the total loss is backpropagated to update the frame-level feature extraction backbone network, the three projection heads, and the three back-projection matrices. The updated frame-level visual feature vector is then returned to step 200 to recalculate the gating, thus forming a closed loop of feature decoupling, prototype cleanup, and boundary correction.

[0107] In an optional embodiment, after obtaining the frame-level visual feature vectors, a pre-trained continuous sign language recognition model is used to perform forward inference on a small sample sign language video sequence. Based on the annotation type and the model's predicted probability distribution, one method is selected from CTC alignment, DTW alignment, and sliding segment mean clustering to generate initial frame-level pseudo-labels and initial boundary region confidence scores for the small sample sign language video sequence. The sliding segment mean clustering method works by sliding a preset sliding window length across the frame-level visual feature sequence frame by frame, calculating the frame-level values ​​of all frames within the window at each window position. The arithmetic mean of the visual feature vectors is used as the segment feature of the window. Then, K-segment clustering with temporal continuity constraints is performed on the window segment features, ensuring that each cluster corresponds to a temporally continuous window interval. After clustering, the intervals are mapped to corresponding sign language words in the word-level label sequence according to their temporal order, where K represents the number of sign language words in the word-level label sequence. The initial centers of the K-means clustering are obtained by dividing the window segment features into K continuous segments according to the word-level label order and then taking the arithmetic mean within each segment. Clustering iteration stops when the cluster affiliation of the window segment features no longer changes. Each frame is assigned a pseudo-label to the corresponding sign language category based on the clustering of its window. The frame position where the clustering of adjacent windows switches is used as the segment boundary position. The preset sliding window length is determined by dividing the product of the average frame rate of the training video and the average duration of the sign language word by two, and rounding the result to a positive integer number of frames. If the rounding result is less than one frame, one frame is taken, so that each window covers half the duration of a sign language word. When using CTC alignment, if there are frames assigned as whitespace characters in the optimal path, no sign language category pseudo-label is assigned to whitespace character frames; instead, they are... A blank frame is marked separately. Blank frames usually appear between adjacent sign language words. The initial boundary region confidence of a blank frame is set to one, so that it tends to be identified as a transition frame in subsequent scoring and does not participate in the initial category prototype construction of any category. The initial boundary region confidence value is in a closed interval between zero and one, indicating the degree to which the frame is located at the boundary of adjacent segments. Its value is obtained by inverse mapping of the distance from the adjacent segment switching position to the frame in the alignment path through a normalization function. The closer the frame is to the segment switching position, the closer the value is to one. The farther the frame is from the segment switching position, the closer the value is to zero.

[0108] In step 100, frame-level visual feature sequences, word-level label sequences, and model prediction probabilities are used simultaneously as alignment inputs because frame-level visual feature sequences provide visual changes between adjacent frames, word-level label sequences define the order of sign language words, and model prediction probabilities provide category confidence information. CTC alignment is suitable for samples that only have word order but lack frame boundaries, DTW alignment is suitable for samples where the movement speeds of different sign language users are inconsistent, and sliding segment mean clustering is suitable for samples where the prediction probabilities of the pre-trained continuous sign language recognition model are insufficient to form a stable path but the segment features can still be aggregated.

[0109] In an optional embodiment, the alignment method selection rules are as follows: CTC alignment is used when the pre-trained continuous sign language recognition model outputs a class probability sequence including whitespace and the input sample has a word-level label sequence; DTW alignment is used when there is a class reference sequence and the difference in movement speed between different sign language users causes frame order stretching; sliding segment mean clustering is used when the model's predicted probability cannot form a path that satisfies the word-level label order; each video segment uses only one alignment method to generate initial frame-level pseudo-labels, whitespace frame markers, and initial boundary region confidence scores, avoiding the simultaneous reception of pseudo-labels from different alignment methods in the same frame.

[0110] CTC alignment, or Connectionist Temporal Classification Alignment, is used to align frame-level predicted sequences with word-level labeled sequences when frame-level labels are lacking. It works by assuming the model outputs a predicted probability sequence for each sign language category and whitespace character in each frame, and the word-level labeled sequence is the known order of sign language words. The system introduces a many-to-one mapping including whitespace characters, summing the probabilities of all frame-level paths that can be restored to word-level labeled sequences after removing duplicate symbols and whitespace characters. The CTC loss is defined as the negative value of the natural logarithm of this probability sum. The probability of each frame-level path is equal to the product of the predicted probabilities of the path symbols taken in each frame along that path. The optimal path obtained through CTC alignment can assign frame-level pseudo-labels and segment boundary positions to each frame.

[0111] DTW alignment refers to dynamic time warping alignment, which uses dynamic programming to find the alignment path with the minimum cumulative distance between two time series, thereby establishing a correspondence between frames when the lengths and speeds of the two sequences are inconsistent. It is used to align candidate fragment frames with the category reference sequence and obtain frame-level pseudo-labels and fragment boundary positions.

[0112] In an optional embodiment, the category reference sequence is obtained from the semantic frame feature sequence of the corresponding sign language category in the training set. The system first extracts candidate segments from the labeled videos of the same sign language category according to the word-level label boundaries, then resamples each candidate segment according to the preset reference sequence length, and finally takes the arithmetic mean of the frame-level visual feature vectors at the same time position to obtain the category reference sequence of the sign language category. The preset reference sequence length is the median of the distribution of the number of candidate segments of the sign language category in the training set and rounded to a positive integer number of frames. The category reference sequence is only used for DTW alignment and is not used as a cleaned-up category prototype.

[0113] In this embodiment, frame-level pseudo-labels are sign language category labels automatically assigned by the system to a single frame; non-blank frame-level pseudo-labels refer to frame-level pseudo-labels with values ​​of a specific sign language category, excluding whitespace characters in CTC alignment, blank frame markers, and frames that have not been assigned a sign language category; non-blank frame-level pseudo-labels are used for category discrimination loss, semantic sub-feature-assisted category discrimination loss, metric learning loss based on cleaned category prototypes, and cleaned category prototype construction; blank frames and frames that have not been assigned a sign language category do not participate in the above category supervision as non-blank frame-level pseudo-label frames.

[0114] In an optional embodiment, the non-blank frame-level pseudo-labels are automatically obtained from the alignment results of step 100. When using CTC alignment, the pre-trained continuous sign language recognition model outputs the predicted probability of each frame belonging to each sign language category and the blank symbol. The system selects the optimal path with the highest path probability from all frame-level paths that can be reconstructed into word-level label sequences. If the path symbol of the optimal path in a certain frame is a specific sign language category, then that sign language category is used as the non-blank frame-level pseudo-label for that frame; if the path symbol is a blank symbol, then that frame is marked as a blank frame. When using DTW alignment, the system dynamically times-propagates the candidate fragment frame sequence with the category reference sequence formed according to the word-level label order. If a frame is mapped to a reference position for a certain sign language category in the alignment path, then the sign language category corresponding to that reference position is used as the non-blank frame-level pseudo-label for that frame. When using sliding segment mean clustering, the system clusters the features of window segments, maps the clustering to the corresponding sign language category according to the word-level label order, and uses the sign language category corresponding to the clustering of the window covering the frame as the non-blank frame-level pseudo-label for that frame. When the same frame is covered by multiple windows and the window clustering is inconsistent, the clustering of the window covering the frame and whose center is closest to the frame is taken as the frame's category. If the distances are the same, the clustering of the window with the highest predicted category probability by the model is taken as the frame's category.

[0115] For a given sign language category, all frames labeled as belonging to that category are collected based on initial pseudo-labels. The frame-level visual feature vectors of these frames are then arithmetically averaged to obtain the initial category prototype for that sign language category. When there are frames whose predicted category probability corresponding to the initial pseudo-label is less than or equal to a preset initial category prototype selection threshold, the system prioritizes selecting frames whose predicted probability of belonging to the corresponding category is greater than the threshold to participate in the initial category prototype construction. The preset initial category prototype selection threshold is determined based on the statistical mean of the predicted probability distribution of each category on the validation set of the pre-trained continuous sign language recognition model. The minimum value among the mean predicted probabilities of all categories is taken as the lower bound for selection, ensuring that the number of frames selected for each category in the initial category prototype construction is no less than the preset threshold. The minimum number of frames for the initial category prototype is preset to be the lower quartile of the frame distribution of each training sample. When the number of frames for a certain category after threshold screening is less than the preset minimum number of frames for the initial category prototype, frames already labeled as the sign language category are added in descending order of the probability predicted by the model for that sign language category, until the preset minimum number of frames for the initial category prototype is reached or all labeled frames for that sign language category are selected. When the alignment result does not provide any initial pseudo-label frames for that sign language category, the video sequence is divided into corresponding initial time intervals according to the order of the sign language category in the word-level label sequence, and the arithmetic mean of the frame-level visual feature vectors of each frame in the initial time interval is used as the initial category prototype for that sign language category.

[0116] In an optional embodiment, when the same sign language category appears multiple times in the same small sample sign language video sequence in the word-level tag sequence, the system first establishes sign language word instances according to the order of appearance of the category in the word-level tag sequence, and then collects initial pseudo-tag frames for each sign language word instance; multiple sign language word instances belonging to the same sign language category are merged into a candidate frame set for that sign language category during the initial category prototype construction stage, and their respective segment boundary positions are still retained during the boundary region calculation stage; a sign language word instance represents a single occurrence in the word-level tag sequence, and a sign language category represents the category name in the recognition category set, and the two are not used interchangeably in the following description.

[0117] First, initial pseudo-labels, blank frame markers, and initial boundary region confidence scores are obtained from the alignment results. Then, initial category prototypes are obtained from the initial pseudo-labels, thus avoiding the circular dependency that requires cleaning up the category prototypes before judging semantic frames and vice versa. Sample features are generated, including structured features, frame-level visual feature vectors, visual sub-features, initial pseudo-labels, blank frame markers, initial boundary region confidence scores, and initial category prototypes. The sample features output in step 100 are used as input in step 200.

[0118] To avoid confusion between the prototype names of the initial stage and the iteration stage, this embodiment defines the initial category prototype as the zeroth round cleansing category prototype; where k represents the current iteration round, k is an integer greater than or equal to one, and the zeroth round represents the stage in which the initial category prototype is constructed from the initial pseudo-labels; starting from the first iteration, the current reference cleansing category prototype used in the kth round to calculate semantic proximity, transition frame boundary constraint loss, and metric learning loss is the k-first round cleansing category prototype; the current reference cleansing category prototype is not always equal to the initial category prototype; only when k is one is the current reference cleansing category prototype equivalent to the zeroth round cleansing category prototype, that is, the initial category prototype constructed from the initial pseudo-labels.

[0119] Step 200: Calculate semantic frame score, transition frame score and redundant frame score for each frame of the small sample sign language video sequence based on sample features, and convert the semantic frame score, transition frame score and redundant frame score into semantic gating, transition gating and redundant gating respectively.

[0120] For each frame, three soft-state scores are calculated: semantic frame score, transition frame score, and redundant frame score.

[0121] First, the variables involved in the scoring are defined and normalized.

[0122] To determine the weight coefficients and thresholds of subsequent scores in the absence of precise frame-level semantic boundary annotations, instead of relying on a frame-by-frame manually annotated tri-state validation set, weakly annotated proxy tri-state frame groups are constructed using the alignment results obtained in step 100 as a sample source for statistical weight coefficients and thresholds. Alignment segments are frame intervals in the alignment results of step 100 that are continuously assigned to the same sign language category or whitespace symbol. For CTC alignment, alignment segments consist of consecutive non-whitespace categories or consecutive whitespace symbols in the optimal path. For DTW alignment, alignment segments consist of frames continuously mapped to the same sign language category reference sequence position. For sliding segment mean clustering, alignment segments consist of consecutive frames with the same... The clustering window consists of frames; the model predicted class probability is the probability that the class probability output by the pre-trained continuous sign language recognition model in the current frame matches the sign language class corresponding to the initial pseudo-label of the frame; if the current frame is a blank frame or no sign language class pseudo-label is obtained, the model predicted class probability is not calculated for the frame to be used as a proxy semantic frame for selection; the nearest segment switching position is the switching position with the shortest frame order distance between the left and right switching positions of the aligned segment to which the current frame belongs; when there is no switching position on one side, only the existing side is used; when there is only one aligned segment in the current video, each frame in the video is processed as being far away from the segment switching position, and the condition of being close to the segment switching position is not met.

[0123] The proxy semantic frame is a frame within a non-blank aligned segment that is far from the segment switching position and whose predicted class probability is greater than a preset proxy semantic probability threshold. Far from the segment switching position means that the frame distance to the nearest segment switching position is greater than a preset proxy inner area distance threshold. The preset proxy inner area distance threshold is the number of frames obtained by rounding down one-quarter of the length of the aligned segment to which the frame belongs, and at least one frame is included. The proxy non-semantic frame is a frame that is not classified as a proxy semantic frame in the set of frames that can participate in the semantic proxy determination. This includes frames close to the segment switching position, frames whose predicted class probability is less than or equal to the preset proxy semantic probability threshold, blank frames, and frames that have not obtained a sign language category pseudo-label. Therefore, within this determination range, the proxy non-semantic frame is the same as the non-proxy semantic frame. Close to the segment switching position means that the frame distance to the nearest segment switching position is not greater than the preset proxy inner area distance threshold.

[0124] The proxy transition frames are selected from frames whose initial boundary region confidence is greater than the preset proxy transition confidence threshold; the proxy non-transition frames are selected from frames whose initial boundary region confidence is less than or equal to the preset proxy transition confidence threshold; the proxy redundant frames are selected from frames within the pose-maintaining segment whose adjacent pose-maintaining similarity is greater than the preset proxy redundancy similarity threshold and whose unstable motion mutation amount is less than the preset proxy redundancy motion threshold; the proxy non-redundancy frames are the remaining frames; the pose-maintaining segment consists of consecutive frames, and the Euclidean distance between the hand motion features of two adjacent frames within this consecutive frame is less than or equal to the preset pose-maintaining motion threshold, which is the median of the Euclidean distance distribution of the hand motion features of two adjacent frames on the training set; the preset proxy semantic probability threshold is the median of the model prediction class probability distribution of non-blank aligned frames on the training set; the preset proxy transition confidence threshold is the median of the initial boundary region confidence distribution of each frame on the training set; the preset proxy redundancy motion threshold is the median of the unstable motion mutation amount distribution of each frame within the pose-maintaining segment on the training set; the preset proxy redundancy similarity threshold is the median of the adjacent pose-maintaining similarity distribution of each frame within the pose-maintaining segment on the training set.

[0125] When calculating the weight coefficients based on the proxy three-state frames, if the proxy state frame set or proxy non-state frame set of a certain scoring type is empty, the discrimination of the corresponding dimension in that scoring type is zero; if the discrimination of all dimensions of that scoring type is zero, the preset weight coefficients of that scoring type are all given the same weight with a sum of one. Thus, the statistics of each weight coefficient and threshold are all based on the proxy grouping automatically derived from word-level labels and alignment results, which is consistent with the application premise of small sample size, weak labeling, and lack of frame-level boundary labeling.

[0126] In an optional embodiment, when additional frame-level labeled samples are available, the system merges the frame-level labeled samples into the weakly labeled proxy tri-state frame group; the frame type of the frame-level labeled samples and the semantic frames, transition frames and redundant frames of the proxy tri-state frame group use the same names, and the statistical weight coefficients and thresholds are calculated according to the merged frame set.

[0127] The confidence score of the key hand region is calculated by adding the frame-level hand detection confidence score to the value of one minus the frame-level occlusion ratio and then dividing by two. Subsequently, the confidence score of the key hand region is processed by a normalization function, and its statistical minimum and maximum values ​​are taken from the global distribution on the training set.

[0128] The confidence score of the hand detection at the frame level is combined with the confidence score of the key hand region by subtracting the frame level occlusion ratio. This is because the hand region is only suitable as a reliable input for semantic frame scoring when it simultaneously meets the two conditions of being detected and not being occluded.

[0129] In the k-th iteration, the semantic proximity between the frame and the prototype of the cleaned category in the k-th minus one iteration is calculated as follows: the cosine similarity between the frame-level visual feature vector of the frame and the prototype of the cleaned category in the k-th minus one iteration of the category corresponding to the pseudo-label of the frame is calculated, and then the cosine similarity is processed by a normalization function. The minimum and maximum values ​​are taken from the current video sequence. For blank frames that have not been assigned any sign language category pseudo-labels, since there is no prototype of the cleaned category in the k-th minus one iteration of the category corresponding to the pseudo-label of the frame, its semantic proximity is taken as the maximum value of the cosine similarity between the frame-level visual feature vector of the frame and the prototypes of the cleaned category in the k-th minus one iteration of all sign language categories. Then the normalization function is processed, so that blank frames have a definite semantic proximity value without relying on category pseudo-labels, and since they are usually not close to the prototype of the cleaned category in the k-th minus one iteration of any sign language category, the semantic proximity value is close to zero.

[0130] The unstable motion mutation is calculated as follows: t represents the current frame number. For any current frame from the second frame to the T-1 frame, the hand motion feature of the (t+1)-th frame is subtracted from twice the hand motion feature of the t-th frame, and then the hand motion feature of the (t-1)-th frame is added. The resulting vector is then subjected to the L2 norm. This motion mutation is used to characterize the degree of motion acceleration of the frame relative to the preceding and following frames. For the first frame of a video without a preceding frame and the last frame of a video without a following frame, the system takes the corresponding value of its adjacent second-first or second-last frame for its unstable motion mutation. When T equals two, the unstable motion mutation of both frames is taken as the L2 norm of the difference vector of the hand motion features of the two frames. When T equals one, the unstable motion mutation of the frame is zero. Subsequently, a normalization function is applied to the unstable motion mutation, and its statistical minimum and maximum values ​​are taken from the global distribution on the training set.

[0131] Semantic proximity and unstable motion mutation are used as category consistency input and motion mutation input, respectively, because the semantic frame should be close to the corresponding category cleansing category prototype and the motion change should conform to the internal continuity of sign language actions; the two, together with the confidence of key hand regions, are included in the semantic frame score, so that the score simultaneously reflects category affiliation, spatial visibility and temporal motion state.

[0132] The semantic frame score is used to indicate whether the frame carries stable sign language category information. It is calculated by multiplying the normalized confidence score of the key hand region by a preset first semantic weight coefficient, adding the normalized semantic proximity multiplied by a preset second semantic weight coefficient, and then subtracting the normalized unstable motion mutation amount multiplied by a preset third semantic weight coefficient. The preset first, second, and third semantic weight coefficients are determined by statistically analyzing the discriminative power between proxy semantic frames and proxy non-semantic frames in the three dimensions of key hand region confidence, semantic proximity, and unstable motion mutation amount, respectively, on the aforementioned weakly labeled proxy tri-state frame grouping. Specifically, the absolute value of the difference between the mean value of proxy semantic frames and the mean value of proxy non-semantic frames in each dimension is calculated, and the three absolute values ​​are normalized to a weight allocation where the sum equals one. The normalized proportion for each dimension is the preset weight coefficient for that dimension. The semantic frame score increases with increasing key hand region confidence, increases with increasing semantic proximity, and decreases with increasing motion mutation amount.

[0133] The transition frame scoring integrates the changes in the average features of the front and back windows, the uncertainty of category assignment, and the confidence of the initial boundary region. The window length is set as the preset transition scoring window frame number, which is determined by dividing the product of the average frame rate of the training video and the average duration of the sign language words by the preset window segmentation number. The preset window segmentation number is determined by first generating integer candidates that cover a range of one-quarter to one-half of the duration of a single sign language word, and then selecting the integer candidate that maximizes the absolute value of the difference between the mean changes in the front and back window features of the proxy transition frame and the proxy non-transition frame in the training set. If multiple integer candidates yield the same absolute value, the integer that makes the window length close to one-third of the duration of a single sign language word is selected. The preset transition scoring window frame number is rounded to a positive integer after being determined in the above manner; if the rounding result is less than one frame, one frame is taken.

[0134] The average feature is a vector obtained by taking the arithmetic mean of the frame-level visual feature vectors of each frame in a frame set along the same dimension, and its dimension is the same as that of the frame-level visual feature vectors. The average feature of the front window is the average feature of the frame-level visual feature vectors of each frame in the consecutive windows before the current frame, excluding the current frame. The average feature of the back window is the average feature of the frame-level visual feature vectors of each frame in the consecutive windows after the current frame, excluding the current frame. For frames near the beginning of the video, if the number of frames that can be taken in the front window is less than the window length, only the frames that actually exist before that frame are used for the arithmetic mean. For frames near the end of the video, if the number of frames that can be taken in the back window is less than the window length, only the frames that actually exist after that frame are used for the arithmetic mean. When there are no frames before the first frame of the video or no frames after the last frame of the video, the average feature of the missing side of the window is taken as the frame-level visual feature vector of that frame itself, so that the change in the front and back window features of the first and last frames of the sequence has a definite value.

[0135] The average features of the front window and the average features of the back window use the frame-level visual feature vectors on both sides of the current frame as input because sign language word switching usually manifests as a change in the feature direction of the segments before and after the current frame; taking the arithmetic mean of the frames within the window can suppress the influence of single-frame detection fluctuations on the scoring of transition frames, making the scoring focus on changes at the segment level.

[0136] The calculation method for the change in features between the front and rear windows is to subtract the window term from one, where the window term is the cosine similarity between the average features of the front window and the average features of the rear window; the calculation method for the uncertainty in class classification is to subtract the maximum probability from one, where the maximum probability is the maximum probability among the probabilities predicted by the pre-trained continuous sign language recognition model that the frame belongs to each sign language category; the calculation method for the transition frame score is to multiply the change in features between the front and rear windows by a preset first transition weight coefficient after normalization, add the uncertainty in class classification after normalization by a preset second transition weight coefficient, and add the confidence of the initial boundary region in the sample features multiplied by a preset third transition weight coefficient; where the preset third transition weight coefficient is... The first transition weight coefficient, the second preset transition weight coefficient, and the third preset transition weight coefficient are determined by statistically analyzing the discriminative power between the proxy transition frame and the proxy non-transition frame on three dimensions: the change in features before and after the window, the uncertainty of category classification, and the confidence of the initial boundary region on the aforementioned weakly labeled proxy three-state frame grouping. The specific method is the same as the method for determining the weight coefficients in semantic frame scoring, that is, calculating the absolute value of the difference between the mean value of the proxy transition frame and the mean value of the proxy non-transition frame on each dimension and normalizing it to a weight allocation whose sum equals one. The transition frame score increases with the increase of the difference in features before and after the window, increases with the increase of category uncertainty, and increases with the increase of the confidence of the initial boundary region.

[0137] Redundant frame scoring indicates whether a frame is a duplicate of its neighboring frames and has insufficient semantic and boundary contributions. First, neighbor-preserving similarity is calculated by adding the cosine similarity between the current frame and the previous frame to the cosine similarity between the current frame and the next frame, then dividing by two. For the first frame of a video without a preceding frame, neighbor-preserving similarity is taken as its cosine similarity with the next frame. For the last frame of a video without a following frame, neighbor-preserving similarity is taken as its cosine similarity with the previous frame. When T equals one, neighbor-preserving similarity is taken as one. To ensure the input of redundant frame scoring is consistent with the final gating, semantic gating is first obtained from semantic frame scoring using the threshold shift and slope amplification method described later, and transition gating is obtained from transition frame scoring before calculating redundant frame scoring. The calculation method for redundant frame scoring is to take the neighbor-preserving similarity... After processing by the normalization function, the weight is multiplied by a preset first redundancy weight coefficient, plus a value minus the semantic gating multiplied by a preset second redundancy weight coefficient, plus a value minus the transition gating multiplied by a preset third redundancy weight coefficient. The preset first redundancy weight coefficient, preset second redundancy weight coefficient, and preset third redundancy weight coefficient are determined by statistically analyzing the distinguishability of proxy redundant frames and proxy non-redundant frames in three dimensions: neighbor-preservation similarity, semantic gating inversion value, and transition gating inversion value, on the aforementioned weakly labeled proxy three-state frame grouping. The specific method is the same as the method for determining the weight coefficients in the semantic frame scoring. This scoring takes semantic gating and transition gating as inputs, so that the attitude-preserving frame carrying stable semantics has a reduced redundancy frame score due to the increase of the semantic gating value, and is therefore not judged as a redundant frame.

[0138] After obtaining three soft-state scores, these scores are threshold-shifted and slope-amplified before being processed by the Sigmoid function to obtain semantic gating, transition gating, and redundancy gating. Specifically, for each type of score, the preset gating threshold corresponding to that type of score is first subtracted, then multiplied by the preset gating slope coefficient corresponding to that type of score, and finally processed by the Sigmoid function. Semantic gating is equal to the semantic frame score minus the preset semantic gating threshold, multiplied by the preset semantic gating slope coefficient, and then processed by the Sigmoid function. The weight of a frame as a semantic frame is represented by: Transition gating, which is equal to the transition frame score minus a preset transition gating threshold, multiplied by a preset transition gating slope coefficient, and then processed by a Sigmoid function; Redundancy gating, which is equal to the redundant frame score minus a preset redundancy gating threshold, multiplied by a preset redundancy gating slope coefficient, and then processed by a Sigmoid function; The preset gating threshold is introduced such that frames with scores equal to the threshold are gated at 0.5, frames with scores greater than the threshold are gated towards 1, and frames with scores less than the threshold are gated towards... To achieve zero, a preset gating slope coefficient is introduced to amplify the differences in scores near the threshold, creating a separation between values ​​tending towards one and zero on both sides of the threshold. This prevents the gating values ​​from concentrating around 0.5 and failing to distinguish between semantic and non-semantic frames. The three preset gating thresholds are respectively taken as the median of the corresponding score distribution on the training set. The preset gating slope coefficient is set such that the difference between the gating mean of the set of scores above the threshold and the gating mean of the set of scores below the threshold on the training set reaches 0.8. The set of scores above the threshold consists of frames with scores greater than the preset gating threshold of that type. The set of frames with scores below the threshold is the set of frames with scores not exceeding the preset gating threshold for this class. The preset gating slope coefficient is determined by searching the score distribution on the training set using a binary search method. The search start and end range is from one to twenty. The search termination condition is that the absolute deviation of the difference between the gating mean of the set of frames with scores above the threshold and the gating mean of the set of frames with scores below the threshold and 0.8 is less than 0.01. If there is no value that meets the search termination condition within the search start and end range, the preset gating slope coefficient that minimizes the absolute deviation within that range is selected. The values ​​of the three gatings are all located in the open interval between zero and one.

[0139] For example, the first The semantic frame score, transition frame score, and redundant frame score of a frame are denoted as follows: , and Its calculation formula can be written as , , ;in, Indicates the first Confidence of key hand regions after frame normalization Indicates the first In the first iteration Semantic proximity after frame normalization Indicates the first The amount of unstable motion abrupt change after frame normalization. Indicates the first The change in window features before and after frame normalization. Indicates the first Uncertainty in class assignment after frame normalization Indicates the first Confidence of the initial boundary region of the frame. Indicates the first Frames maintain similarity after normalization. Indicates the first Semantic gating of frames Indicates the first Frame transition gating , and These represent the preset first semantic weight coefficient, the preset second semantic weight coefficient, and the preset third semantic weight coefficient, respectively. , and These represent the preset first transition weight coefficient, the preset second transition weight coefficient, and the preset third transition weight coefficient, respectively. , and These represent the preset first redundancy weight coefficient, the preset second redundancy weight coefficient, and the preset third redundancy weight coefficient, respectively.

[0140] For example, the transformation from three types of scoring to three types of gating can be uniformly represented as: ,in Representing three types of states: semantic, transitional, or redundant. Indicates a relationship of belonging. Indicates the first Frame in state The lower gate, Indicates the first Frame in state The rating below, Representing state The corresponding preset gate threshold, Representing state The corresponding preset gate slope coefficient, This represents the Sigmoid function.

[0141] When discrete frame types are required, frames with semantic gating greater than or equal to a preset semantic gating value are designated as semantic frames, frames with transition gating greater than or equal to a preset transition gating value are designated as transition frames, and frames with redundancy gating greater than or equal to a preset redundancy gating value are designated as redundant frames. The preset semantic gating value, preset transition gating value, and preset redundancy gating value are collectively referred to as preset gating values. The preset gating values ​​differ from the aforementioned preset gating thresholds. The preset gating thresholds apply to soft-state scoring and are used to generate gating, while the preset gating values ​​apply to the gating output and are used to obtain discrete frame types. If the system only participates in training optimization, discretization is not forced, and semantic gating, transition gating, and redundancy gating are directly used as soft weights for the corresponding training task. Transition frames represent frames located in adjacent sign language word switching regions, whose transition gating is made greater than or equal to the preset transition gating value after transition frame scoring and gating. Redundant frames represent duplicate contribution frames that are similar to adjacent frames, whose redundancy gating is made greater than or equal to the preset redundancy gating value after redundancy scoring and gating.

[0142] The preset gating decision value is set as follows: For weakly labeled proxy tri-state frame groups in the training set, candidate gating values ​​within the zero-to-one closed interval are traversed for semantic gating, transition gating, and redundancy gating, respectively, with a step size of 0.01 for each candidate gating value. For a certain type of gating and its candidate gating values, frames with a gating value greater than or equal to the candidate gating value are classified as frames of that type of state, and frames with a gating value less than the candidate gating value are classified as frames of a different type of state. The arithmetic mean of the proxy state frame recall rate and the proxy non-state frame recall rate is calculated. The frame with the highest value of this arithmetic mean is then selected. Candidate gating values ​​are used as the preset gating values ​​for this type. If multiple candidate gating values ​​have the same arithmetic mean, the candidate gating value that minimizes the absolute value of the difference between the number of determined state frames and the number of agent state frames is selected. If multiple candidate gating values ​​still exist, the median of these candidate gating values ​​is selected. When the set of agent state frames or the set of agent non-state frames for this type is empty, the median of the gating distribution for this type on the training set is selected as the preset gating value for this type. After training, the preset gating values ​​remain consistent during the verification, inference, and online update phases.

[0143] In an optional embodiment, when the set of score frames above or below a certain score distribution in the training set is empty, the preset gating slope coefficient is set to one, and the preset gating threshold is still the median of that score distribution; if all values ​​of that score distribution are the same, the gating outputs 0.5 in the initial stage of training, and the relevant parameters are updated by subsequent joint optimization and gating coordination constraints.

[0144] To prevent the same frame from being used simultaneously as a semantic frame and a redundant frame with a gating weight close to one, the system sets a gating coordination constraint loss. The gating coordination constraint loss is calculated by summing the gating coordination loss terms for frames 1 to T of each small sample sign language video sequence within the current training batch. The gating coordination loss term for each frame is equal to the sum of two gating over-limit penalty terms. The first gating over-limit penalty term is the sum of the semantic gating and the redundant gating minus one, with the maximum value being zero. A penalty is incurred when the sum of the semantic gating and the redundant gating exceeds one, thus preventing the same frame from being used simultaneously as a semantic frame and a redundant frame with a gating weight close to one. The second gating over-limit penalty term is... The sum of semantic gating and transition gating, minus one and then subtracting the preset gating relaxation coefficient, is multiplied by zero to obtain the maximum value. A penalty is incurred when the sum of semantic gating and transition gating exceeds one plus the preset gating relaxation coefficient. Since the trigger threshold for this term is one plus the preset gating relaxation coefficient, the trigger threshold is greater than one when the relaxation coefficient is positive. This ensures that no penalty is incurred when the sum of semantic gating and transition gating is close to one, thus allowing for a natural degree of overlap between semantic frames and transition frames in actual sign language videos. The preset gating relaxation coefficient is determined based on the proportion of frames that are simultaneously classified as proxy semantic frames and proxy transition frames in the aforementioned weakly labeled proxy tri-state frame grouping. This proportion is used as the relaxation coefficient.

[0145] Step 200 assigns different types of frames to different training tasks. Semantic gating input step 300 is used to clean up category prototype construction, transition gating input step 300 is used for transition boundary traction, and redundancy gating input step 300 is used for noise reduction. Gated coordination constraint loss input step 400 participates in joint optimization.

[0146] Step 300: Construct a cleanup category prototype using semantic gating weighting, construct a transition frame boundary constraint loss using transition gating, and calculate the category discrimination loss based on redundant gating, as detailed below. Figure 2 As shown.

[0147] For a certain sign language category, frame-level features are extracted from video samples belonging to that sign language category, and the k-th round of purification category prototypes are constructed using the semantic gating weighting of step 200. In this embodiment, purification refers to retaining the feature contributions of stable semantic frames according to semantic gating when constructing the purification category prototype, and reducing the feature contributions of transition frames, pause frames, and redundant frames according to semantic gating. Purification does not mean deleting candidate frames, nor does it mean generating another type of prototype different from the purification category prototype.

[0148] The calculation method for the k-th round purification category prototype is as follows: For all candidate frames in the k-th iteration with pseudo-labels for the sign language category, the frame-level visual feature vector of each frame is multiplied by the semantic gating of that frame, summed, and then divided by the accumulated term to obtain the k-th round weighted average term. The accumulated term is the sum of the semantic gating of all candidate frames plus a preset constant. When the number of candidate frames with pseudo-labels for the sign language category in the current small sample task or the current batch is greater than or equal to the preset lower limit of the stable frame number of the purification category prototype, and the sum of the semantic gating of its candidate frames is greater than or equal to the preset lower limit of the stable weight of the purification category prototype, the weighted average term is taken as the k-th round purification category prototype. When the number of candidate frames is zero or the sum of the semantic gating of the candidate frames is zero, the k-th round purification category prototype for the sign language category follows the k-1 round purification category prototype. When the number of candidate frames is greater than zero and the sum of the semantic gating of the candidate frames is greater than zero, and the number of candidate frames or the sum of the semantic gating of the candidate frames for the sign language category in the k-th iteration is greater than zero, the k-th round purification category prototype is taken as the k-th round purification category prototype. When the value is less than the corresponding preset stability lower limit, the weighted average term of the k-th round is calculated first, and then the purification category prototype of the k-th minus one round is linearly fused with the weighted average term of the k-th round according to the fusion coefficient to obtain the purification category prototype of the k-th round. The preset stability lower limit includes the preset stability frame number lower limit and the preset stability weight lower limit of the purification category prototype. Specifically, the number of candidate frames is less than the preset stability frame number lower limit or the sum of semantic gates is less than the preset stability weight lower limit of the purification category prototype. The fusion coefficient is the minimum value between the quotient obtained by dividing the sum of semantic gates of candidate frames by the preset stability weight lower limit of the purification category prototype and one. The weight of the weighted average term of the k-th round during fusion is the fusion coefficient, and the weight of the purification category prototype of the k-th minus one round is one minus the fusion coefficient. The preset stability frame number lower limit and the preset stability weight lower limit of the purification category prototype are respectively taken as the lower quartile of the distribution of the number of candidate frames participating in purification in each round of each category on the training set and the distribution of the sum of semantic gates.

[0149] For example, sign language categories can be categorized as , will the In the round of iteration, the pseudo-label is the sign language category. The set of candidate frames is denoted as The number of candidate frames is denoted as The sum of the semantic gating of the candidate frames is denoted as , will the The round weighted average term is denoted as ,but ;in, This represents the condition separator in set construction. Indicates the first Frame-level pseudo-tags Represents a set Number of inner frame numbers, summation symbol This indicates summation based on the set of frame numbers specified by the subscript. Represents a set All frame numbers Summing corresponding terms Indicates the first Frame-level visual feature vectors of a frame. Indicates the first Semantic gating of frames This represents a preset constant used for purifying the prototype accumulation item.

[0150] For example, the first The prototype of the wheel purification category is denoted as , will the The prototype of the reduced purification category is denoted as The lower limit of the stable frame rate of the preset purification category prototype is denoted as: The lower limit of the stable weight of the preset purification category prototype is denoted as... The fusion coefficient is denoted as Then the first The prototype of the wheel purification category can be represented as: The conditions are and ; can be represented as The conditions are or In other cases, it can be expressed as ; where function symbols This function represents the minimum value and outputs the minimum value among the input values. and These correspond to the aforementioned lower limit of stable frame rate for the prototype of the preset purification category and the lower limit of stable weight for the prototype of the preset purification category, respectively. and subscript This indicates that the corresponding parameter is a lower bound parameter, and does not indicate a function call. Indicates the first Wheel sign language categories The prototype fusion coefficient.

[0151] In an optional embodiment, when the prototype of the purification category in the k-th minus one round does not exist and the current batch cannot generate the k-th round weighted average term for the sign language category, the system reads the historical purification category prototype from the historical semantic frame feature cache of the sign language category in the training set as the k-th round purification category prototype; if the historical semantic frame feature cache does not contain the category record, then the category does not participate in the purification category prototype update, metric learning loss, and transition frame boundary constraint loss in the current batch; the historical semantic frame feature cache is a set of frame-level visual feature vectors that have passed semantic gating and are saved for each sign language category during the training phase, and the historical purification category prototype is the category prototype obtained by semantically gating the frame-level visual feature vectors in this set.

[0152] Unlike the approach of taking the arithmetic average of all frames in a video segment, semantic gating weighting is used, so that frames with semantic gating values ​​close to one dominate the sanitization category prototype, and the contributions of transition frames, pause frames, and redundant frames to the sanitization category prototype are reduced according to their semantic gating. The sanitization category prototype of the k-th round serves as the current reference sanitization category prototype for the next round of transition boundary traction, metric learning, and semantic proximity calculation, and also serves as the input for the cross-sign language few-shot task.

[0153] The transition gating output from step 200 is used to construct the transition frame boundary constraint loss. During the training phase, this loss is not only calculated for discrete transition frames, but also for each frame from frame 1 to frame T of the small sample sign language video sequence within the current training batch, and these boundary loss terms are summed. For frame t, based on the alignment result obtained in step 100 or the frame-level pseudo-label sequence of the current iteration, the system searches forward from frame t minus one as the previous category, and searches backward from frame t plus one as the next category for the nearest non-blank sign language category that is different from the previous category. When there is no non-blank sign language category forward or backward, or when two different non-blank sign language categories cannot be obtained from forward or backward searches, the t-th frame does not constitute a valid transition candidate frame, and its boundary loss term is zero. At the discrete frame type level, the transition frame determined by step 200 according to the preset transition gating judgment value is not used as the core feature of category discrimination, nor is it included in the purification category prototype update. At the boundary loss calculation level, the t-th frame that constitutes a valid transition candidate frame only determines the weight of the boundary loss term through its transition gating, and does not change its discrete frame type or purification category prototype update rule due to participating in the transition frame boundary constraint loss calculation.

[0154] For frame t, which constitutes a valid transition candidate frame, reference semantic frames are selected for the previous category and the next category, respectively. The length of the preset neighborhood window of frame t is twice the number of preset transition scoring window frames in step 200, that is, the neighborhood range is formed by taking the preset number of transition scoring window frames forward and backward from frame t. Among the frames belonging to the previous category within the preset neighborhood window, the frame with the largest semantic gating value is selected as the reference semantic frame for the previous category. Among the frames belonging to the next category within the preset neighborhood window, the frame with the largest semantic gating value is selected as the reference semantic frame for the next category. When there are no frames belonging to the corresponding category on one side of the preset neighborhood window, the frame with the largest semantic gating value is selected as the reference semantic frame among all frames belonging to the corresponding category in the current video. When there are still no frames belonging to the corresponding category in the current video, the current reference purification category prototype of the corresponding category is used as the reference semantic feature, and the Euclidean distance from the reference semantic feature to the current reference purification category prototype of the corresponding category is set to zero.

[0155] The transition frame boundary constraint loss is calculated as follows: First, the boundary interval penalty term of frame t for the previous category is calculated. This boundary interval penalty term is equal to the maximum value among the following values ​​and zero: the Euclidean distance from the frame-level visual feature vector of the previous category reference semantic frame to the prototype of the current reference cleaned category of the previous category, plus a preset boundary interval threshold, and then the Euclidean distance from the frame-level visual feature vector of frame t to the prototype of the current reference cleaned category of the previous category is subtracted. Then, the boundary interval penalty term of frame t for the next category is calculated in the same way, that is, the previous category and the previous category reference semantic frame are replaced with the next category and the next category reference semantic frame, respectively. The boundary loss term of frame t is equal to the transition gate of this frame multiplied by the sum of the boundary interval penalty terms of the previous category and the boundary interval penalty terms of the next category. The boundary loss terms of each video sequence from frame 1 to frame T in the current training batch are summed to obtain the transition frame boundary constraint loss of this training batch.

[0156] The preset boundary interval threshold is determined based on the statistical distribution of the Euclidean distance between the sign language category purification category prototypes in the training set. Before the first iteration, the preset boundary interval threshold is calculated based on the purification category prototypes in the zeroth round. The preset boundary interval threshold remains unchanged in subsequent iterations of the current training task. It is obtained by multiplying the mean of the Euclidean distances between all different category purification category prototypes by a preset scaling factor. This preset scaling factor is determined by searching on the validation set. The search range is 0.05 to 0.5, with a step size of 0.05. The selection criterion is the maximum scaling factor value where, under this scaling factor value, the transition frame boundary constraint loss decreases to less than one-tenth of the initial value after training convergence, and at the same time, the mean of the Euclidean distance from the same type of frame features to the same type of purification category prototype in the training set does not exceed 1.2 times the mean value when the transition frame boundary constraint is not applied. If there is no scaling factor within the search range that simultaneously meets the above selection criteria, the sum of the normalized default amount of the transition frame boundary constraint loss decrease ratio and the increase in the distance of the same type of frame features corresponding to each scaling factor is calculated, and the scaling factor with the smallest sum is selected. The feature distance uses Euclidean distance, which is calculated by subtracting two feature vectors element by element and taking the L2 norm of the difference vector, i.e., the arithmetic square root of the sum of the squares of the elements of the difference vector. This constraint requires that the Euclidean distance from the frame-level visual feature vector of frame t to the current reference cleansing class prototype of a certain adjacent category is not less than the Euclidean distance from the reference semantic frame of the adjacent category to the same current reference cleansing class prototype plus a preset boundary interval threshold. When the distance from frame t to the current reference cleansing class prototype of a certain adjacent category is no less than this threshold, the penalty for the corresponding category is zero, and the frame is no longer pushed further away. Since the previous category uses the reference semantic frame of the previous category as a reference and the next category uses the reference semantic frame of the next category as a reference, this constraint only requires that the effective transition candidate frame maintains an interval relative to the semantic frame of each adjacent category itself, so that frames with a transition gate greater than or equal to the preset transition gate judgment value are between two sign words and are not absorbed into the category's internal frame by the cleansing class prototype of any adjacent category. At the same time, the region with a transition gate greater than or equal to the preset transition gate judgment value is used as the sign word switching candidate boundary.

[0157] For example, the set of valid transition candidate frames within the current training batch can be denoted as... Let the Euclidean distance function be denoted as ,in and Let represent two eigenvectors to be compared; for the ... frame, Indicates the previous category, Indicates the latter category, This represents the frame-level visual feature vector of the previous category reference semantic frame. This represents the frame-level visual feature vector of the reference semantic frame for the latter category. This indicates the current reference purification category prototype of the previous category. This indicates the current reference purification category prototype for the latter category. Indicates the preset boundary interval threshold; according to the first The penalty is applied when the Euclidean distance from the current reference cleanup class prototype of the adjacent class does not reach the distance of the adjacent class reference semantic frame plus a preset boundary interval threshold. The boundary interval penalty term of the previous class can be expressed as:

[0158] ;

[0159] The second category boundary margin penalty term can be expressed as:

[0160] ;

[0161] Among them, function symbols This function represents the maximum value and outputs the maximum value among the input values. Indicates the first The boundary spacing penalty term for the previous category of the frame. Indicates the first The boundary spacing penalty term for the next category of frames.

[0162] For example, the first The boundary loss term of the frame is denoted as The transition frame boundary constraint loss of the current training batch is denoted as... ,but , ;in, Indicates the first Frame transition gating; for frames that do not constitute valid transition candidate frames, their boundary loss terms are set to zero according to the aforementioned rules and are not included. .

[0163] In an optional embodiment, when two adjacent sign language word instances in a word-level label sequence have the same sign language category, the former category and the latter category are no longer distinguished solely by the sign language category name in the transition frame boundary constraint loss, but by the order in which the sign language word instances appear. In this case, the two adjacent sign language word instances share the cleaned category prototype of that sign language category, but each selects a reference semantic frame within its own segment, so that there are still sign language word switching candidate boundaries between repeated sign language words of the same category.

[0164] For redundant frames determined in step 200, they are not directly deleted, but their contribution to class discrimination loss is reduced and used for adjacent frame consistency correction; the redundancy weighting coefficient is calculated by subtracting the redundancy gate of the frame from one.

[0165] The calculation method for the category discrimination loss includes the following: when frame-level pseudo-labels exist, the category discrimination loss is weighted by a redundancy reduction coefficient. The calculation method is to sum the values ​​of frames with non-blank frame-level pseudo-labels in the current training batch. The contribution of each frame is equal to the redundancy reduction coefficient of that frame multiplied by the cross-entropy category discrimination term of that frame. The cross-entropy category discrimination term of that frame is equal to the natural logarithm of the probability corresponding to the frame-level pseudo-label in the category probability output by the temporal modeling network for that frame, and then the negative value. Blank frames and frames that have not obtained sign language category pseudo-labels do not have cross-entropy category discrimination terms calculated, and their category discrimination loss contribution is zero.

[0166] In an optional embodiment, when only word-level sequence labels or sentence-level labels are available, the category discrimination loss is taken as the sequence-level category discrimination loss. The sequence-level category discrimination loss calculates the CTC loss by comparing the frame-level predicted sequence output by the model with the sign language word sequence corresponding to the video. The calculation method of CTC loss follows the aforementioned CTC alignment definition, that is, summing the probabilities of all frame-level paths that can be restored to the sign language word sequence, taking the natural logarithm of the sum of probabilities, and then taking the negative value. Thus, the category discrimination loss is reflected as the redundancy-weighted cross-entropy category discrimination loss under the frame-level pseudo-label condition, and as the CTC category discrimination loss under the condition of only word-level sequence labels or sentence-level labels.

[0167] Two types of class discriminant loss are used to make the same training process compatible with frame-level pseudo-labels and word-level sequence labels. The redundant weighted cross-entropy class discriminant loss provides frame-by-frame supervision using aligned frames, while the CTC class discriminant loss provides sequence supervision using word order constraints, so that samples lacking frame-level pseudo-labels can still participate in model parameter updates.

[0168] Simultaneously, a neighborhood consistency constraint loss is applied to redundant frames using a redundancy-gated weighted approach. The neighborhood consistency constraint loss is calculated by summing the neighborhood consistency loss terms of the first to T frames of each small sample sign language video sequence within the current training batch. The neighborhood consistency loss term for each frame is equal to the redundancy gating of that frame multiplied by the L2 norm of the difference vector obtained by subtracting the arithmetic mean of the previous and next frame's frame-level visual feature vectors from the frame-level visual feature vector of that frame. For the first frame of a video without a preceding frame, the difference vector is the first frame's frame-level visual feature vector minus the next frame's frame-level visual feature vector. For the last frame of a video without a following frame, the difference vector is the last frame's frame-level visual feature vector minus the previous frame's frame-level visual feature vector. When T equals one, the difference vector is zero, thus giving the neighborhood consistency constraint of the first and last frames of the sequence and a single frame of video a definite value. This constraint maintains the temporal smoothness of redundant frames, making their features close to the mean of the features of adjacent frames, but they do not participate in the cleanup category prototype-dominated learning.

[0169] The output cleanup category prototype, transition frame boundary constraint loss, category discrimination loss, and neighborhood consistency constraint loss are input into step 400 to participate in joint optimization.

[0170] Step 400: Jointly optimize the category discrimination loss, transition frame boundary constraint loss and metric learning loss based on the cleaned category prototype to obtain the total loss. Update the continuous sign language recognition model with the total loss and iteratively update the cleaned category prototype to obtain the optimized continuous sign language recognition model and the cleaned category prototype.

[0171] The total loss is obtained by jointly optimizing the category discrimination loss, the metric learning loss based on the cleaned category prototype, the transition frame boundary constraint loss, the neighborhood consistency constraint loss, the decoupling orthogonality constraint loss, the sub-feature reconstruction loss, the semantic sub-feature-assisted category discrimination loss, and the gating coordination constraint loss. The total loss is calculated as follows: the category discrimination loss is multiplied by a preset metric learning loss weight parameter, the boundary constraint loss weight parameter is multiplied by the transition frame boundary constraint loss, the neighborhood consistency loss weight parameter is multiplied by the neighborhood consistency constraint loss, the decoupling orthogonality constraint loss weight parameter is multiplied by the decoupling orthogonality constraint loss, the reconstruction loss weight parameter is multiplied by the sub-feature reconstruction loss, the auxiliary discrimination loss weight parameter is multiplied by the semantic sub-feature-assisted category discrimination loss, and the gating coordination loss weight parameter is multiplied by the gating coordination constraint loss. The category discrimination loss, the transition frame boundary constraint loss, and the neighborhood consistency constraint loss are all given in step 300. The decoupling orthogonality constraint loss, the sub-feature reconstruction loss, and the semantic sub-feature-assisted category discrimination loss are all given in step 300. The auxiliary category discrimination loss is jointly provided by the projection head set in step 100, the semantic sub-feature auxiliary classifier, the semantic sub-features, transition sub-features, redundant sub-features in the sample features, and their normalized forms. The gating coordination constraint loss is given in step 200. The above-mentioned preset loss weight parameters are determined in the following way: First, each loss term is trained separately on the validation set until convergence, and the magnitude of the value of each loss term at the time of convergence is recorded. Then, based on the convergence magnitude of the category discrimination loss, the weight parameters of the remaining loss terms are set as the convergence magnitude of the category discrimination loss. The ratio of the level to the convergence magnitude of the corresponding loss term ensures that the contributions of each loss term to the total loss are of the same order of magnitude in the initial stage of joint optimization. Subsequently, a validation set grid search candidate set is constructed with this weight parameter as the center. The candidate set covers the neighborhood of the weight parameter of the same order of magnitude in a logarithmic equidistant manner. The final value is selected with the validation set recognition accuracy as the main indicator and whether the total loss converges as the constraint condition. When multiple candidates achieve the same validation set recognition accuracy, the candidate with the fewest total loss convergence rounds is selected as the final value of the preset loss weight parameter.

[0172] In an optional embodiment, when a loss term fails to converge or converges to zero during individual training, the preset loss weight parameter of the loss term is not directly determined by the magnitude ratio, but rather the median candidate in the candidate set of the validation set grid search is used as the initial candidate, and the final value is selected based on the validation set recognition accuracy and the total loss convergence status; the weight parameter of the loss term that does not participate in the calculation of the current training task remains configured in the training task, but the numerical contribution of the loss term is zero.

[0173] The above losses are jointly optimized because relying solely on the class discrimination loss would push transition frames and repeatable frames into the class, while relying solely on the prototype metric lacks temporal boundary constraints. By simultaneously constraining class discrimination, prototype distance, transition boundary, redundancy smoothing, and sub-feature decoupling through the total loss, model parameter updates can be performed around the same cleansing class prototype target.

[0174] The metric learning loss based on the cleanup category prototype is semantically gated and weighted. It is calculated by summing the metric loss terms of the valid metric frames in the current training batch to obtain the metric learning loss of the training batch. A valid metric frame is defined as a frame with a non-blank frame-level pseudo-label, the pseudo-label category having a current reference cleanup category prototype, and at least one non-target category current reference cleanup category prototype in the current training task. The metric loss terms of blank frames, frames without sign language category pseudo-labels, frames whose pseudo-label categories do not have a current reference cleanup category prototype, and frames without a non-target category current reference cleanup category prototype are set to zero.

[0175] For a valid metric frame, the metric loss term for that frame is equal to the semantic gating of that frame multiplied by the metric interval penalty term. The metric interval penalty term is equal to the maximum of the following values ​​and zero: the Euclidean distance from the frame-level visual feature vector of that frame to the current reference cleansing class prototype of the pseudo-label class of that frame, minus the Euclidean distance from the frame-level visual feature vector of that frame to the nearest non-target class current reference cleansing class prototype, plus the preset metric learning interval. The nearest non-target class current reference cleansing class prototype refers to the cleansing class prototype with the smallest Euclidean distance to the frame-level visual feature vector among all current reference cleansing class prototypes different from the pseudo-label class of that frame in the current training task. The preset metric learning interval... The margin is determined by the difference between the mean Euclidean distance of features from the same class of frames to the prototype of the current cleansing category and the mean Euclidean distance of features from different classes of frames to the nearest non-target class cleansing category prototype. The absolute value of this difference is taken as the basic value of the margin, which is used to reflect the actual separation degree of intra-class distance and inter-class distance in the current feature space. When the current training task only includes one sign language category and there is no current reference cleansing category prototype for non-target categories, there are no effective metric frames in the current training batch, and the metric learning loss based on the cleansing category prototype is not included in the total loss calculation of the training task. This loss makes the frame-level features closer to the prototype of the current cleansing category and farther away from the nearest non-target class cleansing category prototype.

[0176] The decoupling orthogonal constraint loss is based on the normalized semantic sub-features, normalized transition sub-features, and normalized redundant sub-features in the sample features. It is calculated by summing the orthogonal loss terms of the first to T frames of each small sample sign language video sequence within the current training batch. The orthogonal loss term for each frame is equal to the square of the cosine similarity between the normalized semantic sub-features and the normalized transition sub-features, plus the square of the cosine similarity between the normalized semantic sub-features and the normalized redundant sub-features, and so on. Since the three normalized sub-features are all unit-length vectors, the cosine similarity will not exhibit numerical instability near the zero vector. This constraint is used to reduce the directional coupling between the three types of sub-features.

[0177] The sub-feature reconstruction loss uses semantic sub-features, transition sub-features, and redundant sub-features directly output by the projection head without L2 normalization. It is calculated by summing the reconstruction loss terms of the first to Tth frames of each small sample sign language video sequence in the current training batch to obtain the sub-feature reconstruction loss of the training batch. The reconstruction loss term of each frame is equal to the square of the L2 norm of the difference vector between the frame-level reconstructed feature vector and the original frame-level visual feature vector of the frame. The frame-level reconstructed feature vector is obtained by adding the vector of the semantic sub-feature mapped by the semantic backprojection matrix, the vector of the transition sub-feature mapped by the transition backprojection matrix, and the vector of the redundant sub-feature mapped by the redundant backprojection matrix. This loss allows the three sub-features to retain the information of the original frame-level visual features as a whole, thereby avoiding feature collapse into a single value or a zero vector.

[0178] The semantic sub-feature auxiliary category discrimination loss uses the semantic sub-feature auxiliary classifier set in step 100. Its calculation method is to sum the values ​​of frames with non-blank frame-level pseudo-labels in the current training batch. The auxiliary discrimination loss term for each frame is equal to the semantic gating of that frame multiplied by the auxiliary cross-entropy category discrimination term. This auxiliary cross-entropy category discrimination term is equal to the natural logarithm of the probability corresponding to the frame-level pseudo-label in the semantic sub-feature output of the semantic sub-feature auxiliary classifier for that frame, followed by a negative value. Blank frames and frames that have not obtained sign language category pseudo-labels do not have their auxiliary cross-entropy category discrimination term calculated, and their auxiliary discrimination loss term is zero. The semantic sub-feature auxiliary category discrimination loss and the sub-feature reconstruction loss jointly constrain the projection head, ensuring that the decoupled orthogonal constraints do not drive the projection head to output directions without discriminative meaning.

[0179] For example, the set of all frames in the current training batch can be denoted as Let the set of frames with non-blank frame-level pseudo-labels in the current training batch be denoted as:

[0180] ;

[0181] in Let the set of sign language categories be represented; the decoupling orthogonal constraint loss, sub-feature reconstruction loss, and semantic sub-feature-assisted category discrimination loss be denoted as follows: , and ,but:

[0182] ;

[0183] ;

[0184] in, This represents the transpose of a vector. The squared cosine similarity between pairs of features is represented by the squared inner product between normalized sub-features. Reconstructing feature vectors at the frame level With frame-level visual feature vectors The square of the L2 norm represents the reconstruction error.

[0185] For example, let the semantic sub-feature auxiliary classifier be denoted as The semantic sub-features are used to assist the classifier for the first... Sign language categories in frame output The probability is denoted as Then the semantic sub-feature-assisted category discrimination loss can be expressed as: ;in, Indicates the first The frame is used as the semantic sub-feature auxiliary classifier to output the sign language category. The probability, Represents the normalized exponential function, Represents the natural logarithm. Indicates the first Frame-level pseudo-tags for frames.

[0186] The model parameters, projector parameters, and backprojection matrix parameters are updated using backpropagation with the total loss. After each update, the algorithm returns to step 200 to recalculate the score and to step 300 to regenerate the purification category prototype, thus completing the iterative optimization.

[0187] The joint training steps of the continuous sign language recognition model are as follows: Initialize the zeroth round cleanup category prototype and projector parameters; extract cross-sign language small sample tasks from the training set and form training batches; execute steps 100 to 300 on the training batches to obtain various losses and the kth round cleanup category prototype; update the frame-level feature extraction backbone network, temporal modeling network, semantic projector, transition projector, redundant projector, semantic backprojection matrix, transition backprojection matrix, redundant backprojection matrix, and semantic sub-feature auxiliary classifier by backpropagation according to the total loss; after each round of update, fix the parameters of this round and recalculate the gating and cleanup category prototype, and stop training when the recognition accuracy on the validation set stops increasing or reaches the preset number of training rounds; this training step allows the learnable parameters and cleanup category prototype to be updated alternately, avoiding the use of old gating to constrain the frame-level visual feature vectors that have changed over time.

[0188] During the iterative optimization process, video samples from different sign language speakers are divided into support sets and query sets to construct a cross-sign language speaker few-sample task. The support set is used to generate a cleaned-up category prototype in step 300, and the query set is used to verify the model's recognition ability on new sign language speaker samples.

[0189] For each training task, the model generates cleaned-up category prototypes for each sign language category in that task based on the support set. Each video sample in the query set is called a query sample, and the frame-level visual feature vector of the current frame to be judged in the query sample is called the query frame feature. The query frame feature and the frame-level visual feature vector in the sample feature have the same meaning, and their source is the query sample in the query set. The model calculates the category probability for the query frame feature. The category probability is calculated as follows: for a certain sign language category, the natural exponent is calculated by taking the negative Euclidean distance from the query frame feature to the cleaned-up category prototype of that sign language category, and then dividing by the sum of the natural exponents of the negative Euclidean distances from the query frame feature to the cleaned-up category prototypes of all sign language categories in that training task. This probability is obtained by normalizing the negative Euclidean distance from the query frame feature to the cleaned-up category prototypes of each sign language category. Through cross-task iterative training, the model reduces its dependence on the individual sign language performer's movement style, body posture, and background features. The training result is the converged continuous sign language recognition model and the cleaned-up category prototypes for each sign language category, which are input into step 500 for output.

[0190] Step 500: Based on the optimized continuous sign language recognition model and the cleaned category prototype, output the recognition results of a small sample sign language video sequence.

[0191] After training, the output includes: the optimized continuous sign language recognition model; the cleaned-up category prototype for each type of sign language word; the distribution of semantic frames, transition frames, and redundant frames in each sign language video segment, determined by semantic gating, transition gating, redundancy gating, and corresponding preset gating judgment values; the boundary positions or boundary confidence of sign language words determined by transition gating and preset transition gating judgment values; and the recognition results and corresponding confidence scores of the sign language videos.

[0192] The output of semantic frames, transition frames, and redundant frames, as well as the location or confidence level of sign language word boundaries, is to ensure that the recognition results have traceability. When the recognition results require manual correction or online updates, the system can locate the category discrimination criteria, switching regions, and deweighted frames.

[0193] When the system is used for online recognition, a two-stage process is adopted for the input continuous sign language video to eliminate the circular dependence of the scoring on pseudo-labels. In the first stage, the system first executes step 100 to generate sample features. The frame-level visual feature vector in the sample features is given by the frame-level feature extraction backbone network. The initial pseudo-labels, blank frame markers and initial boundary region confidence in the sample features are given by the pre-trained continuous sign language recognition model to perform a forward recognition and alignment of the video, which serves as the source of pseudo-labels required for calculating semantic proximity in step 200. In the second stage, the system performs the three-state frame decoupling in step 200 based on the initial pseudo-labels obtained in the first stage. Then, according to the semantic frame features and the cleaned category prototypes of each sign language category, category matching is performed according to the category probability calculation method in step 400. At the same time, the sign language word switching position is determined according to the transition gating and the preset transition gating judgment value. The influence of non-semantic frames on the recognition result is reduced by the redundancy reduction coefficient, i.e., one minus the redundancy gating. Finally, the recognition result is obtained by the category matching and switching position in the second stage. The confidence of the recognition result is taken as the probability value of the matched category in the category probability calculation method.

[0194] The system identifies consecutive frame regions with a transition gating value not less than a preset transition gating value as candidate switching regions according to frame order, and selects the frame with the largest transition gating value within each candidate switching region as the segment segmentation position. Based on the segment segmentation positions, the small sample sign language video sequence is divided into multiple segments to be identified. For each segment to be identified, the system calculates the sum of the product of the category probability and the semantic gating value for the same sign language category for each frame within the segment, and divides it by the semantic gating term, which is the sum of the semantic gating values ​​within the segment plus a preset constant, to obtain the segment category score for that sign language category. The sign language category with the largest segment category score is determined as the identification category of the segment to be identified, and the identification categories are arranged according to the chronological order of the segments to be identified to obtain the identification result of the small sample sign language video sequence.

[0195] In an optional embodiment, online recognition employs a sliding buffer to perform a two-stage process. The sliding buffer stores the frame-level visual feature vectors, initial pseudo-labels, blank frame markers, and initial boundary region confidence scores of the current video segment and the preset number of cached frames before and after it. When a new frame enters the buffer, the system only recalculates the three-state frame decoupling and category matching for the cached segments affected by the new frame, while the recognition results that have already been output and are not within the buffer range remain unchanged. The preset number of cached frames is the maximum value between the preset number of transition scoring window frames and the preset agent inner region distance threshold.

[0196] Upon receiving a manually corrected result or a recognition result with a confidence level exceeding a preset incremental update confidence threshold, the cleansing category prototype for the corresponding sign language category is updated online with weighted average. The preset incremental update confidence threshold is determined based on the probability distribution of the category corresponding to the correct recognition result output by the model on the validation set. The threshold is taken as the probability value at the 25th percentile of the cumulative distribution function of the probability of the correct recognition result, ensuring that the cleansing category prototype update is triggered only when the recognition confidence reaches the probability level within the top three-quarters of the correct recognition results on the validation set. The updated cleansing category prototype is calculated by multiplying the preset historical prototype retention coefficient by the cleansing category prototype before the update, adding one and subtracting the preset historical prototype retention coefficient, and then multiplying by the weighted average of the semantic frame features of the new sample. The preset historical prototype retention coefficient is determined based on the ratio of the number of historical samples accumulated to the number of new samples for the current category, and the quotient of the number of historical samples divided by the sum of the number of historical samples and the number of new samples is used as the retention coefficient. When the historical sample... When the proportion of this quantity in the sum of the historical sample quantity and the new sample quantity increases, the proportion of the previous cleansing category prototype in the updated cleansing category prototype also increases; when this proportion decreases, the proportion of the weighted average of the semantic frame features of the new sample increases; the weighted average of the semantic frame features of the new sample is calculated as follows: in the new sample, select a set of frames with pseudo-labels of the sign language category and semantic gating greater than or equal to the preset semantic gating screening threshold, multiply the frame-level visual feature vector of each frame by the semantic gating of that frame, sum them, and then divide by the set term, which is the sum of the semantic gating of all frames in the set plus a preset constant; when the set of frames is empty or the sum of the semantic gating of all frames in the set is zero, the cleansing category prototype update is not performed, and a correction prompt is output; the preset semantic gating screening threshold is determined based on the semantic gating distribution of all semantic frames after convergence during the training phase, and the median of the distribution is taken as the screening threshold to ensure that the semantic gating of the frames selected for update is greater than or equal to the median of the distribution.

[0197] In an optional embodiment, when the manual correction result indicates that the sign language category corresponding to the original recognition result is incorrect, the system does not use the original recognition result to update the cleaned category prototype of the original category, but uses the manually corrected sign language category as the target category to re-execute the weighted average of the semantic frame features of the new sample; if the manual correction result only gives the segment-level category and not the frame-level boundary, the system uses the semantic gating obtained in step 200 to filter the semantic frame, and the boundary position is still determined by the transition gating and the preset transition gating judgment value.

[0198] The results of manual correction and the recognition results with confidence levels exceeding the preset incremental update confidence threshold are used as inputs for online updates because these two types of inputs correspond to manually confirmed samples and model self-confirmed samples, respectively. Using preset historical prototype retention coefficients for weighted updates allows the number of historical samples and the number of new samples to jointly determine the update magnitude of the purification category prototype.

[0199] The system updates the cleanup category prototype only using frames with semantic gating greater than or equal to a preset semantic gating threshold, avoiding the introduction of non-semantic noise in subsequent incremental training. The confidence score for hand detection in video segments used for rejection is the minimum of the frame-level hand detection confidence scores within that segment; the occlusion ratio is the maximum of the frame-level occlusion ratios within that segment; and the boundary uncertainty is the maximum of the transition gating values ​​within that segment. For video segments with hand detection confidence less than or equal to a preset hand detection rejection threshold, occlusion ratios exceeding a preset occlusion rejection threshold, or boundary uncertainties exceeding a preset uncertainty rejection threshold, the system does not update the cleanup category prototype, only outputting a prompt for correction or waiting for manual correction. The preset hand detection rejection threshold is the threshold set when a hand detection bounding box is correctly detected in the training set. The lower quartile of the hand detection confidence distribution is used, and the upper quartile of the occlusion ratio distribution of proxy semantic frames in the training set is used as the preset occlusion rejection threshold. The preset uncertainty rejection threshold is determined based on the transition gating distribution of proxy transition frames and the transition gating distribution of proxy semantic frames in the training set. When the density curves of the two distributions have a unique intersection point, the transition gating value corresponding to that intersection point is taken as the rejection threshold. When the density curves of the two distributions do not have an intersection point or have multiple intersection points that are difficult to uniquely determine in small sample scenarios, the upper quartile of the transition gating distribution of proxy semantic frames is taken as the rejection threshold, so that frames exceeding this threshold are statistically more likely to be classified as proxy transition frames than as proxy semantic frames. The updated cleanup category prototype is used as the cleanup category prototype for the next stage of online recognition and incremental optimization, and together with the above recognition output, constitutes the final result of this system.

[0200] Example 2:

[0201] See Figure 3 As shown, a few-shot model optimization system based on feature decoupling is provided for executing the aforementioned few-shot model optimization method based on feature decoupling. The system includes:

[0202] The feature representation module 101 is used to receive small sample sign language video sequences and generate sample features for the small sample sign language video sequences based on a pre-trained continuous sign language recognition model.

[0203] The decoupled scoring module 102 calculates semantic frame scores, transition frame scores, and redundant frame scores for each frame of a small sample sign language video sequence based on sample features, and converts the semantic frame scores, transition frame scores, and redundant frame scores into semantic gating, transition gating, and redundant gating, respectively.

[0204] The purification and weight reduction module 103 is used to construct a purification category prototype with semantic gating weighting, construct the transition frame boundary constraint loss with transition gating, and calculate the category discrimination loss based on redundant gating.

[0205] The loss optimization module 104 is used to jointly optimize the category discrimination loss, the transition frame boundary constraint loss and the metric learning loss based on the cleaned category prototype to obtain the total loss. The total loss is used to update the continuous sign language recognition model, and the cleaned category prototype is iteratively updated to obtain the optimized continuous sign language recognition model and the cleaned category prototype.

[0206] The recognition output module 105 outputs the recognition results of a small sample sign language video sequence based on the optimized continuous sign language recognition model and the cleaned category prototype.

[0207] The embodiments of this application have been described above, but these embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments based on the guidance of these embodiments, and all of them are within the protection scope of these embodiments.

Claims

1. A few-sample model optimization method based on feature decoupling, characterized in that, Includes the following steps: Receive a small sample of sign language video sequences and generate sample features for the small sample of sign language video sequences based on a pre-trained continuous sign language recognition model; Based on the sample features, semantic frame scores, transition frame scores, and redundant frame scores are calculated for each frame of a small sample sign language video sequence, and the semantic frame scores, transition frame scores, and redundant frame scores are converted into semantic gating, transition gating, and redundant gating, respectively. A purification category prototype is constructed using semantic gating weighting, a transition frame boundary constraint loss is constructed using transition gating, and a category discrimination loss is calculated based on redundant gating. The total loss is obtained by jointly optimizing the category discrimination loss, the transition frame boundary constraint loss and the metric learning loss based on the cleaned category prototype. The continuous sign language recognition model is updated with the total loss, and the cleaned category prototype is iteratively updated to obtain the optimized continuous sign language recognition model and the cleaned category prototype. The optimized continuous sign language recognition model and the cleaned-up category prototype output the recognition results of a small sample of sign language video sequences.

2. The few-sample model optimization method based on feature decoupling according to claim 1, characterized in that, Methods for generating sample features include: For each frame of a small sample sign language video sequence, the hand detection model and the human pose detection model are called to obtain the detection results. The occlusion ratio is calculated based on the detection results, and structured features are generated based on the detection results and the occlusion ratio. A pre-trained continuous sign language recognition model is used to perform forward inference on a small sample of sign language video sequences to generate initial pseudo-labels, blank frame markers, and initial boundary region confidence scores. Initial category prototypes are then constructed from the initial pseudo-labels. The frame-level visual feature vector is output by the backbone network of the continuous sign language recognition model, and visual sub-features are extracted based on the frame-level visual feature vector. Generate sample features, which include structured features, frame-level visual feature vectors, visual sub-features, initial pseudo-labels, blank frame markers, initial boundary region confidence, and initial category prototypes.

3. The few-sample model optimization method based on feature decoupling according to claim 2, characterized in that, Methods for calculating the occlusion ratio include: The detection results are organized into anatomical left-hand instances and anatomical right-hand instances, and the instance corresponding to any anatomical hand is called the target hand instance. The first criterion is obtained by calculating the proportion of the number of key points with a key point confidence level less than the preset key point visibility threshold in the detection results of the target hand instance to the total number of key points of the target hand instance. The second criterion is calculated by the ratio of the area of ​​the hand detection box in the detection results of the target hand instance to the median area of ​​the hand detection box of the same anatomical hand. The third criterion is calculated based on the mean confidence of key points in the overlapping area of ​​the hand detection boxes of the anatomical left hand instance and the anatomical right hand instance; The maximum value among the first criterion, the second criterion, and the third criterion is taken as the occlusion ratio.

4. The few-sample model optimization method based on feature decoupling according to claim 3, characterized in that, Methods for obtaining semantic gating, transition gating, and redundancy gating include: Calculate semantic frame scores based on sample features; Transition frame scores are calculated based on frame-level visual feature vectors, the probability of predicting sign language categories using a continuous sign language recognition model, and the confidence level of the initial boundary region. The semantic frame score and the transition frame score are respectively processed by threshold shifting and slope amplification, and then processed by the Sigmoid function to obtain semantic gating and transition gating respectively. The neighbor-preserving similarity is calculated based on the cosine similarity of adjacent frames in a small sample sign language video sequence. The redundant frame score is calculated based on the neighbor-preserving similarity, semantic gating, and transition gating. The redundant frame scores are processed by threshold shifting and slope amplification, and then processed by the Sigmoid function to obtain the redundancy gate.

5. The few-sample model optimization method based on feature decoupling according to claim 4, characterized in that, Methods for calculating semantic frame scores include: The confidence level of key hand regions is calculated based on the hand detection confidence level and occlusion ratio in the detection results. Semantic proximity is calculated based on the cosine similarity between frame-level visual feature vectors and the prototype of the previous round of purification categories. The unstable motion mutation amount is calculated based on the hand motion features in the structured features. The semantic frame score is obtained by weighting and combining the confidence of the key hand region, semantic proximity and unstable motion mutation amount.

6. The few-sample model optimization method based on feature decoupling according to claim 4, characterized in that, Methods for constructing purification category prototypes include: For all candidate frames with pseudo-labels of the same sign language category in the k-th iteration, the frame-level visual feature vector of each frame is multiplied by the semantic gate, summed, and then divided by the cumulative term to obtain the weighted average term of the k-th iteration; where the cumulative term is the sum of the semantic gates of all candidate frames plus a preset constant, k is an integer greater than or equal to one, and the zero-th round purification category prototype is the initial category prototype. When the number of candidate frames in the k-th iteration is zero or the sum of the semantic gating of the candidate frames is zero, the prototype of the cleanup category for the sign language category in the k-th iteration is the prototype of the cleanup category in the k-th minus one iteration. When the sum of the number of candidate frames and the semantic gating is greater than zero, and the number of candidate frames or the sum of the semantic gating of candidate frames in the sign language category in the k-th iteration is less than the corresponding preset stability lower limit, the purification category prototype of the k-th minus one-th round is linearly fused with the weighted average term of the k-th round to obtain the purification category prototype of the k-th round. Otherwise, take the weighted average term of the kth round as the prototype of the purification category of the kth round.

7. The few-sample model optimization method based on feature decoupling according to claim 6, characterized in that, Methods for calculating the boundary constraint loss of transition frames include: Based on the initial pseudo-label, different preceding and following categories are determined for each frame, and frames that simultaneously exist in both preceding and following categories are considered as valid transitional candidate frames. Within the preset neighborhood window of the effective transition candidate frames, the frame with the largest semantic gating on both the previous and next categories is taken as the reference semantic frame for the corresponding category. Based on the effective transition candidate frames, the frame-level visual feature vectors of the reference semantic frames of the previous and next categories, and the current reference cleanup category prototypes of the previous and next categories, the boundary margin penalty terms of the previous and next categories are calculated; the current reference cleanup category prototype is the cleanup category prototype of the k-th round. Based on the transition gating of effective transition candidate frames and the boundary interval penalty terms between the previous and next categories, the boundary loss term is calculated. The multiple boundary loss terms are accumulated to obtain the transition frame boundary constraint loss.

8. The few-sample model optimization method based on feature decoupling according to claim 7, characterized in that, Methods for obtaining the total loss include: The learning loss is calculated based on semantic gating, frame-level visual feature vectors, and the current reference cleanup category prototype. The neighborhood consistency constraint loss is calculated by weighting the difference between the frame-level visual feature vector and the mean of the frame-level visual feature vector of its neighboring frames using redundancy gating. A gate coordination constraint loss is constructed based on semantic gating, redundant gating, and transition gating; Based on visual sub-features, we calculate the decoupling orthogonal constraint loss, sub-feature reconstruction loss, and semantic sub-feature-assisted category discrimination loss. The total loss is calculated based on the category discrimination loss, metric learning loss, transition frame boundary constraint loss, neighborhood consistency constraint loss, decoupling orthogonality constraint loss, sub-feature reconstruction loss, semantic sub-feature-assisted category discrimination loss, and gating coordination constraint loss.

9. The few-sample model optimization method based on feature decoupling according to claim 8, characterized in that, Methods for calculating the decoupling orthogonal constraint loss, sub-feature reconstruction loss, and semantic sub-feature-assisted category discrimination loss include: Visual sub-features include semantic sub-features, transition sub-features, and redundant sub-features; The decoupling orthogonal constraint loss is obtained by summing the squares of the cosine similarities between each pair of normalized semantic sub-features, normalized transition sub-features, and normalized redundant sub-features. The semantic sub-features, transition sub-features, and redundant sub-features are mapped by their respective back-projection matrices and then added together to obtain the frame-level reconstructed feature vector. The sub-feature reconstruction loss is obtained by summing the squared L2 norm of the difference vector between the frame-level reconstructed feature vector and the frame-level visual feature vector. Based on the category probabilities and pseudo-labels output by semantic sub-features, calculate the semantic sub-feature-assisted category discrimination loss.

10. A few-sample model optimization system based on feature decoupling, characterized in that, The system is used to execute the feature-decoupling-based few-sample model optimization method as described in any one of claims 1-9; the system includes: The feature representation module is used to receive small sample sign language video sequences and generate sample features for the small sample sign language video sequences based on a pre-trained continuous sign language recognition model. The decoupled scoring module calculates semantic frame scores, transition frame scores, and redundant frame scores for each frame of a small sample sign language video sequence based on sample features, and converts the semantic frame scores, transition frame scores, and redundant frame scores into semantic gating, transition gating, and redundant gating, respectively. The purification and weight reduction module is used to construct purification category prototypes with semantic gating weighting, construct transition frame boundary constraint loss with transition gating, and calculate category discrimination loss based on redundant gating. The loss optimization module is used to jointly optimize the category discrimination loss, the transition frame boundary constraint loss and the metric learning loss based on the cleaned category prototype to obtain the total loss. The total loss is used to update the continuous sign language recognition model, and the cleaned category prototype is iteratively updated to obtain the optimized continuous sign language recognition model and the cleaned category prototype. The recognition output module outputs the recognition results of small sample sign language video sequences based on the optimized continuous sign language recognition model and the cleaned category prototype.

Images