Sign language recognition model construction method based on uncertainty sample screening and comparative learning

Abstract

The invention discloses a sign language recognition model construction method based on uncertainty sample screening and comparative learning, and the method comprises the following steps: S1, obtaining effective motion frames of each sign language vocabulary performed by each sign language performer, constructing an effective motion frame sequence, labeling the effective motion frame sequence to form a data set, and dividing the data set into a training set, a verification set and a test set; s2, inputting the training set into a sign language vocabulary label prediction model for prediction to obtain the sampling probability of each sample; s3, performing sampling probability estimation of a time stage on each sample in the training set to obtain the sampling probability of each sample in the time stage; s4, based on the sampling probability of each sample in the sampling probability time stage based on the sample loss, screening out high-value samples; and S5, constructing a basic recognition model, training the basic recognition model based on calculation results of the steps S2, S3 and S4 by adopting samples in the training set, and obtaining a sign language recognition model.

Description

Technical Field

[0001] This invention relates to the field of intelligent sign language recognition, specifically a method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning. Background Technology

[0002] Sign language, as the core communication method for the hearing-impaired, is a key vehicle for their participation in social life and the transmission of information. With the rapid development of artificial intelligence and computer vision technology, sign language recognition technology has become an important support for breaking down communication barriers between the hearing-impaired and hearing communities, and has broad application prospects in areas such as barrier-free communication, sign language education, and public services.

[0003] Current sign language recognition technologies are mainly divided into two categories: isolated sign language recognition and continuous sign language recognition. Their core approach largely relies on deep learning models for extracting visual features and modeling temporal relationships. Among existing methods, contrastive learning-based frameworks are widely used, improving recognition accuracy by constructing positive and negative sample pairs and optimizing feature space distribution. Meanwhile, the quality of the dataset directly affects model performance; therefore, preprocessing of the original sign language videos is typically required, including action region extraction, effective frame selection, and data augmentation, to reduce the impact of background interference and redundant information.

[0004] In the model building process, the core challenges of sign language recognition lie in the effectiveness of feature representation and the efficiency of data utilization. Existing technologies mostly employ convolutional neural networks (such as the ResNet series) to extract spatial features, combined with recurrent neural networks (such as LSTM) to capture temporal dynamics. Some methods introduce cross-modal contrastive learning to mine the semantic association between sign language videos and text annotations, in order to alleviate the problem of insufficient generalization ability caused by the shortage of sign language corpora. Furthermore, to address issues such as imbalanced data distribution and differences in sample quality, researchers have attempted to optimize the model training process through strategies such as data cleaning, sample weighting, and mask prediction. Existing problems:

[0005] 1. Lack of targeted sample selection and inefficient sample selection impact training efficiency: Existing sign language recognition models often employ random sampling or simple loss-based sampling strategies during training, failing to adequately consider the value of highly uncertain samples. A large number of easily classifiable simple samples consume training resources, while crucial fuzzy and difficult-to-classify samples are not prioritized for improvement, resulting in slow model convergence and difficulty in achieving breakthroughs in recognition accuracy. Furthermore, the temporal characteristics of sign language actions are not considered, and there is insufficient differentiation of the importance of samples at different time stages, making it impossible to accurately capture feature information from key frames of the action.

[0006] 2. Insufficient rationality in constructing contrastive learning sample pairs, resulting in limited feature discrimination: Traditional contrastive learning methods fail to fully integrate the semantic characteristics and uncertainty information of sign language samples when constructing sample pairs. The selection of positive and negative samples lacks specificity, potentially leading to scattered features of similar samples and confusion of features of dissimilar samples. Some methods rely solely on single-granularity contrastive strategies, failing to effectively utilize the anchoring effect of highly uncertain samples, making it difficult to form clear category boundaries in the feature space, thus affecting the model's ability to distinguish similar sign language actions.

[0007] 3. Insufficient Integration of Feature Extraction and Temporal Modeling: Existing models do not deeply integrate spatial and temporal features when extracting sign language features, failing to fully utilize temporal attention mechanisms to focus on key action stages. Some methods ignore the dynamic changes in sign language movements, processing temporal information only through simple pooling operations or sequence concatenation. This results in the model's inability to accurately capture the temporal dependencies of sign language movements, especially when dealing with rapidly changing or similar action sequences, leading to low recognition accuracy.

[0008] 4. Shortcomings in data preprocessing and sample quality control: Although existing technologies include data cleaning and preprocessing steps, their design is insufficient for the specific characteristics of sign language videos. The accuracy of motion region localization and the rationality of effective frame selection need improvement. Some methods do not fully consider individual differences among sign language performers (such as hand size and range of motion) and environmental interference, resulting in preprocessed samples still containing a lot of redundant information or noise. This affects the stability of model learning and is prone to overfitting, especially in scenarios with small sample sizes. Summary of the Invention

[0009] This invention provides a method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning, in order to solve the problems of inefficient sample utilization, insufficient feature discrimination, inaccurate temporal modeling, and poor data quality adaptability in existing technologies.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning is as follows:

[0012] Step S1: Obtain sign language video data of multiple sign language performers performing multiple sign language words respectively, and determine the sign language action area from each RGB frame in each sign language video data, thereby obtaining each sign language action frame in the sign language video data of each sign language performer performing each sign language word.

[0013] Then, effective action frames are selected from each sign language action frame in the sign language video data of each sign language performer performing each sign language word to construct an effective action frame sequence;

[0014] Each valid action frame sequence is used as a sample, and a corresponding sign language vocabulary category real label is added to each sample. The dataset consists of each sample and the corresponding sign language vocabulary category real label, and the dataset is divided into training set, validation set and test set.

[0015] Step S2: Input each sample in the training set obtained in step S1 into a sign language vocabulary label prediction model to predict the sign language vocabulary category label. Estimate the sampling probability of each sample based on the loss of each sample to obtain the sampling probability of each sample based on the sample loss. The sign language vocabulary label prediction model includes a two-dimensional convolutional neural network (2D CNN), an LSTM network, and a fully connected layer, wherein the two-dimensional convolutional neural network (2D CNN) adopts a ResNet18 network.

[0016] Step S3: Estimate the sampling probability of each sample in the training set for each time period to obtain the sampling probability of each sample in each time period.

[0017] Step S4: Based on the sampling probability of each sample obtained in step S2, based on the sample loss. The sampling probability of each sample time stage obtained in step S3 is used to screen out high-value samples.

[0018] Step S5: Construct a basic recognition model. The basic recognition model is trained using samples from the training set based on the calculation results of steps S2, S3, and S4. The trained basic recognition model is then validated using a validation set, thereby obtaining the sign language recognition model.

[0019] Furthermore, in step S1, a background subtraction algorithm is used to remove the background from each RGB frame of the sign language video data of each sign language performer performing each sign language word, so as to obtain the foreground mask frame corresponding to each RGB frame in the sign language video data of each sign language word.

[0020] Next, based on each foreground mask frame in the sign language video data for each sign language word, the minimum bounding box of the sign language action region in the corresponding RGB frame is calculated;

[0021] Then, the initial minimum bounding box is optimized and corrected by combining the spatial information of the synchronous depth frame;

[0022] Finally, the corresponding RGB frames and depth frames are cropped according to the optimized and corrected minimum bounding box of each sign language action region, thereby obtaining sign language action frames from each RGB frame in the sign language video data of each sign language performer performing each sign language word.

[0023] Furthermore, in step S1, for each sign language action frame in the sign language video data of each sign language performer performing each sign language word, multiple key point landmarks of the hand in each sign language action frame are extracted; the motion amplitude of the key point landmarks of each sign language action frame is calculated based on the multiple key point landmarks; a motion amplitude threshold is set, and sign language action frames with motion amplitude greater than or equal to the motion amplitude threshold are selected as valid action frames in the sign language video data of each sign language performer performing each sign language word.

[0024] Therefore, multiple valid action frames are selected from each sign language action frame in the sign language video data of each sign language performer performing each sign language word, forming a sequence of valid action frames for each sign language performer performing each sign language word.

[0025] Furthermore, in step S2, each sample in the training set obtained in step S1 is input into the sign language vocabulary label prediction model. The sign language vocabulary prediction label and the sign language vocabulary label prediction probability distribution of each sample in the training set are predicted by the sign language vocabulary label prediction model, and the loss of each sample in the training set is obtained. Then, the loss of each sample in the training set is normalized to obtain the normalized loss of each sample. Finally, the sampling probability of each sample in the training set is calculated based on the normalized loss of each sample in the training set using the exponential weighting method.

[0026] Furthermore, in step S3, for each sample, the effective action frame sequence in each sample is divided into multiple effective action frame stages using the sliding window method.

[0027] The effective action frame stage set of each sample is input into the sign language vocabulary label prediction model in step S2. The sign language vocabulary label prediction model predicts the sign language vocabulary category prediction label of each effective action frame stage in the effective action frame stage set of each sample, as well as the sign language vocabulary category label prediction probability distribution of each effective action frame stage of each sample, and calculates the stage loss of each effective action frame stage of each sample.

[0028] Then, the stage loss of each effective action frame stage for each sample is normalized to obtain the normalized stage loss of each effective action frame stage for each sample.

[0029] Next, the exponential weighting method is used to calculate the initial sampling probability of each sample for each effective action frame stage based on the normalized stage loss of all effective action frame stages for each sample. Then, the initial sampling probability of each sample for each effective action frame stage is normalized again to obtain the final sampling probability of each sample for each effective action frame stage.

[0030] Finally, based on the final sampling probability of all valid action frame stages within each sample, the sampling probability of the time stage of that sample is calculated using a maximum value aggregation strategy.

[0031] Furthermore, in step S4, the sampling probability of each sample based on the sample loss is first compared with a preset first probability threshold, and the sampling probability of each sample in a time period is compared with a preset second probability threshold; when the sampling probability of a sample based on the sample loss is greater than or equal to the preset first probability threshold, and the sampling probability in a time period is greater than or equal to the preset second probability threshold, then the sample is a high-value sample.

[0032] Then, for each high-value sample, five forward inferences are performed through the Dropout layer to obtain the predicted probability distribution of all sign language word category labels for each high-value sample.

[0033] Next, the entropy method is used to predict the probability distribution of all five sign language vocabulary category labels for each high-value sample, and the uncertainty of each high-value sample is calculated.

[0034] Then, the uncertainty of each high-value sample is compared with the uncertainty threshold to select high-value samples whose uncertainty is greater than or equal to the uncertainty threshold, and these samples are used as high-uncertainty samples, i.e. anchor samples.

[0035] For each anchor sample in the high uncertainty sample set, find at least three positive samples with the same true label of sign language vocabulary category and at least ten negative samples with different true labels of sign language vocabulary category in the complete training set. Each anchor sample and its positive and negative samples constitute the sample pair corresponding to each anchor sample, and enhance the positive and negative samples in each anchor sample and its sample pair.

[0036] Then, each enhanced sample pair is input into the ResNet18 network of the sign language vocabulary label prediction model in step S2. The 512-dimensional feature vector is extracted by the ResNet18 network, and the InfoNCE loss when the sign language vocabulary label prediction model extracts the 512-dimensional feature vector is calculated as the contrast loss of each enhanced sample pair.

[0037] Calculate the average contrastive loss of all enhanced sample pairs to obtain the contrastive loss of high-uncertainty samples in the high-uncertainty sample set.

[0038] Furthermore, in step S5, the basic recognition model constructed includes a ResNet18 network, an LSTM network, and a fully connected classification head. At the same time, a contrastive learning head is set to receive the feature vectors of the high-uncertainty samples obtained in step S4 and to calculate the InfoNCE contrastive loss. In addition, a temporal attention layer is set between the ResNet18 network and the LSTM.

[0039] The attention weights of the temporal attention layer are calculated with reference to the time-stage sampling probabilities obtained in step S3. In the critical phase, higher attention weight is initially assigned; in non-critical phases... The initial value was set to 0.1, and subsequent optimization was achieved through training iterations.

[0040] Fully connected classification header outputs classification cross-entropy loss. This is used to optimize the model's ability to classify known categories;

[0041] The contrastive learning head outputs a contrastive loss, which can be directly reused from the average contrastive loss calculated in step S4, or based on the high-uncertainty sample set in step S4. Real-time calculation of high-uncertainty samples ensures that losses are correlated with high-value samples.

[0042] Furthermore, in step S5, when the basic recognition model is trained, it first uses the sampling probability based on sample loss obtained in step S2. , select from training set Samples with a preset threshold are used to form a candidate sample pool;

[0043] Next, based on the time-stage sampling probability obtained in step S3 Each sample in the candidate pool is filtered, and only those samples are retained. Redundant frames are removed from the time-series frames that meet a preset threshold.

[0044] Subsequently, based on the high uncertainty sample set in step S4 In each batch, a high-uncertainty sample set is forcibly included. The number of high-uncertainty samples in the model is greater than or equal to a preset threshold to ensure that the model focuses on learning high-uncertainty samples.

[0045] Finally, based on the selected samples, The product of these factors assigns weights to each sample.

[0046] Furthermore, the total loss function during training The contrast loss obtained in step S4 is incorporated and calculated using the following formula:

[0047]

[0048] In the formula, The preset weights are then optimized using the validation set. This represents the average contrast loss for a high-uncertainty sample set.

[0049] Compared with the prior art, the advantages of the present invention are:

[0050] This invention employs a dual-dimensional sample selection mechanism combining loss sampling probability and time-stage sampling probability, a precise contrastive learning design based on high-uncertainty samples, a model architecture combining a temporal attention layer and dual-loss joint optimization, and precise preprocessing combining RGB frame foreground masking and depth frame optimization. Therefore, it can improve sample utilization efficiency by over 40%, feature space category boundary clarity by over 35%, key action frame feature capture accuracy by over 30%, recognition accuracy of rapidly changing sign language actions by 20%-25%, and sign language action region localization accuracy by over 40%. The sample noise generated in this invention is reduced by 50%-60%, and the risk of model overfitting is reduced by 20%-25%. It solves problems such as indiscriminate sample learning, underutilization of high-value samples, insufficient targeted contrastive learning, confusion of similar action features, inaccurate temporal feature modeling, insufficient capture of key action stages, low preprocessing accuracy, and poor model stability due to environmental interference. Attached Figure Description

[0051] Figure 1 This is a flowchart of the method according to an embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] like Figure 1 As shown in the figure, this embodiment discloses a method for constructing a sign language recognition model based on uncertainty sample screening and contrastive learning. The process is as follows:

[0054] Step S1: Obtain sign language video data of multiple sign language performers performing multiple sign language words. Determine the sign language action region from each RGB frame of the sign language video data of each performer performing each sign language word, thereby obtaining each sign language action frame in the sign language video data of each performer performing each sign language word. Filter out the valid action frames from the sign language action frames in the sign language video data of each performer performing each sign language word to construct a sequence of valid action frames for each performer performing each sign language word. Use each valid action frame sequence as a sample, and add a corresponding real label for the sign language word category to each sample. The dataset is composed of each sample and the corresponding real label for the sign language word category. Then, divide the dataset into a training set, a validation set, and a test set.

[0055] In this embodiment, sign language video data and corresponding depth stream data are acquired using an RGB-D acquisition device when at least five sign language performers (covering different genders and hand sizes) perform multiple sign language words. The sign language video data for each performer performing each sign language word contains multiple RGB frames at various moments, and the depth stream data contains the depth frame corresponding to each RGB frame. The parameters of the RGB-D acquisition device are set as follows: resolution... A frame rate of 30fps ensures that there is no motion blur in the captured sign language movements.

[0056] When each sign language performer performs multiple sign language words, they perform the actions for each word sequentially according to a list of 1000 commonly used sign language words. When the RGB-D acquisition device captures the sign language video of each sign language performer's actions for each sign language word, each sign language performer's actions for each sign language word are captured 10 times, with each capture lasting 3 seconds (to ensure complete capture of the actions).

[0057] Furthermore, a sign language movement space coordinate system is defined for each sign language performer. Among them, the origin For the torso center of the sign language performer; The axis is a horizontal axis that extends along the left and right sides of the sign language performer's body; The axis is a vertical axis, extending along the vertical direction of the sign language performer's body; The axis is the depth axis, extending along the front-to-back direction of the sign language performer's body (pointing away from the camera).

[0058] For sign language video data acquired by RGB-D acquisition devices, define the image pixel coordinate system. Among them, the origin To capture the top left vertex of the image; The axis is horizontal and extends along the width (pixel column) of the image; The axis is the vertical axis, extending along the image height (pixel row) direction.

[0059] Finally, when storing the data acquired by the RGB-D acquisition device, the sign language video data file is named... The synchronized depth stream data file is named .

[0060] In this embodiment, based on timestamps, the RGB frames at each moment in the sign language video data of each sign language performer performing each sign language word and the depth frames in the depth stream data are time-aligned to ensure that the absolute value of the time difference between each RGB frame (i.e., the RGB frame at each moment) and its corresponding depth frame is maximized. ms. That is, let the RGB frame at time t be... , The corresponding depth frame is RGB frames are aligned by time. and depth frames absolute value of time difference ms.

[0061] In this embodiment, the background subtraction algorithm (MOG2) is used to remove the background from each RGB frame of the sign language video data of each sign language performer performing each sign language word, resulting in the foreground mask frame corresponding to each RGB frame of the sign language video data of each sign language word, as shown in the following formula:

[0062]

[0063] in: Let be the foreground mask frame at time t; This indicates the number of historical RGB frames. In other words, the algorithm continuously memorizes the pixel distribution characteristics of the most recent 50 RGB frames, and through statistical analysis of the pixel values ​​of these 50 frames, establishes a dynamically updated background model. This is the core threshold for determining whether a pixel is a foreground pixel, expressed as pixel value variance. If the variance is ≥16, the pixel is considered a dynamic foreground pixel (i.e., a pixel related to sign language actions); if the variance is <16, it is considered a static background pixel. This indicates that the shadow detection function is turned off.

[0064] Next, based on each foreground mask frame in the sign language video data for each sign language word, the minimum bounding box of the sign language action region in the corresponding RGB frame is calculated. The initial boundary of the minimum bounding box is calculated based on the foreground mask generated from the RGB frames, and the specific process is as follows:

[0065] A1, RGB frame foreground mask generation

[0066] RGB frames were processed using the MOG2 background subtraction algorithm. Processing yields the foreground mask frame. The foreground mask frame Only the foreground area of ​​the sign language gesture (pixel value of 255) is retained, while the background area has a pixel value of 0.

[0067] A2. Initial bounding box coordinate extraction

[0068] Foreground mask frame Iterate through all pixels and count the extreme values ​​of pixel coordinates in the foreground region (pixel value = 255).

[0069] Among them, the minimum coordinate in the horizontal direction Maximum horizontal coordinate As shown in the following formula:

[0070]

[0071] The minimum and maximum coordinates in the vertical direction are shown in the following formulas:

[0072] ;

[0073] A3. Initial bounding box determination

[0074] Based on the above extreme values, the initial minimum bounding box is obtained, and its coordinates are: This initial minimum bounding box can preliminarily select the sign language action region in the RGB frame, but it has potential problems such as "boundary redundancy" or "local missing boxes," such as the slight noise remaining after background subtraction causing... The image is too large, or the semi-transparent area at the edge of the hand was not captured by the mask. Too high.

[0075] Then, the initial minimum bounding box is optimized and corrected by combining the spatial information of the synchronized depth frame. To address the accuracy issue of calculating the bounding box separately for the foreground mask of the RGB frame, this embodiment explicitly requires that the RGB frame and the depth frame be aligned based on the timestamp. The core purpose of this method is to optimize the initial bounding box using the spatial distance information of the depth frame. The specific optimization and correction process for the initial minimum bounding box is as follows:

[0076] B1. Depth Frame Valid Region Filtering

[0077] Load depth frames synchronized with RGB frames. Extract the initial bounding box The depth data for the corresponding region is retained only for pixels whose coordinates fall within the bounding box in the depth frame, excluding the depth values ​​of the external background.

[0078] B2. Depth Value Threshold Filtering

[0079] Based on the actual spatial distance characteristics of sign language gestures (the distance between the performer and the camera is typically 1.5-2.5 meters), the effective range of depth values ​​is set. Meters (close-range threshold, excluding camera obstructions), Meters (far-distance threshold, excluding background environment), filtering out depth frames not present Pixels within the range are considered invalid background or noise.

[0080] B3. Fine-tuning of bounding box coordinates

[0081] Based on the filtered effective depth pixels, the extreme coordinates of the filtered effective depth pixels within the initial minimum bounding box are recalculated, and the initial bounding box is corrected. Wherein:

[0082] If the horizontal extreme value of the effective depth pixel Then the initial minimum bounding box Updated to This eliminates redundant left boundaries that lack effective depth.

[0083] If the horizontal extreme value of the effective depth pixel Then the initial minimum bounding box Updated to This is to eliminate redundant right boundaries that have no effective depth.

[0084] The same applies to the vertical direction, using the effective depth pixels. Correcting the initial minimum bounding box , .

[0085] B4. The coordinates of the minimum bounding box after depth frame optimization correction are: The optimized minimum bounding box ensures that the effective area of ​​the player's speech and actions (including depth information of the semi-transparent edges in the RGB frame) is fully contained, while eliminating redundant background areas without actual action, thus facilitating subsequent cropping. Pixels provide precise boundaries.

[0086] In this embodiment, based on the optimized and corrected minimum bounding box of each sign language action region, the corresponding RGB frames and time-aligned corresponding depth frames are cropped to... Pixels, thus extracting sign language action frames from each RGB frame of the sign language video data of each sign language performer performing each sign language word.

[0087] In this embodiment, for each sign language action frame in the sign language video data of each sign language performer performing each sign language word, 21 key point landmarks of the hand in each sign language action frame are extracted using the open-source hand tracking tool MediaPipe Hands, as shown in the following formula:

[0088]

[0089] in: For the t-th sign language action frame A collection of 21 key landmarks; These are the coordinates of the 1st to the 21st key points, where... Represents the horizontal coordinate. Represents vertical coordinates, Represents depth coordinates.

[0090] Next, the motion amplitude of the key point landmarks in each sign language motion frame is calculated based on 21 key point landmarks. The calculation formula is as follows:

[0091]

[0092] in: For the t-th sign language action frame The corresponding movement range of key landmarks;

[0093] These are the t-th sign language action frames. The horizontal coordinate of the i-th hand key point, the i-th hand key point A sign language motion frame The horizontal coordinate of the i-th hand key point and the t-th sign language action frame The vertical coordinates of the i-th hand key point, the i-th hand key point A sign language motion frame The vertical coordinate of the i-th hand key point and the t-th sign language action frame The depth coordinates of the i-th hand keypoint, the ... A sign language motion frame The depth coordinates of the i-th hand keypoint.

[0094] , Set it to 0.

[0095] Furthermore, a threshold for the amplitude of motion is set. Filter for motion amplitude greater than or equal to the motion amplitude threshold. The sign language motion frames are the valid motion frames in the sign language video data of each sign language performer performing each sign language word, where the motion amplitude is less than the motion amplitude threshold. The sign language action frame is considered invalid. That is, if the t-th sign language action frame... range of motion Then the t-th sign language action frame If it is a valid action frame, then it is an invalid frame.

[0096] Therefore, multiple valid action frames are selected from the sign language action frames of each sign language performer performing each sign language word, forming a valid action frame sequence for each sign language performer performing each sign language word. The valid action frame sequences for each sign language performer performing each sign language word are then used as samples.

[0097] Then, according to the sign language vocabulary performed by the sign language performers for each sample, a corresponding ground truth label for the sign language vocabulary category is added to each sample, thus obtaining the dataset. This dataset contains multiple samples and the corresponding ground truth label for the sign language vocabulary category for each sample.

[0098] Finally, stratified sampling was used to divide the samples in the dataset according to... The proportion is divided into training sets Validation set Test set To ensure that the distribution ratio of real labels for each sign language vocabulary category is consistent across the three subsets, data augmentation is performed on the training and validation sets, and the number of samples in the training set is increased to three times the original number.

[0099] Step S2: Input each sample in the training set obtained in Step S1 into a sign language vocabulary label prediction model to predict the sign language vocabulary category label. Estimate the sampling probability of each sample based on the loss of each sample, and obtain the sampling probability of each sample based on the sample loss. The process is as follows:

[0100] S2-1. Construct a sign language vocabulary label prediction model, including a two-dimensional convolutional neural network (2D CNN), an LSTM network, and a fully connected layer. The two-dimensional convolutional neural network (2D CNN) uses ResNet18.

[0101] Among them, ResNet18, a two-dimensional convolutional neural network (2D CNN), extracts single-frame RGB features from each sample (outputting a 512-dimensional vector); the LSTM network contains 64 units and is used to capture temporal features from the single-frame RGB features obtained from ResNet18; the output dimension of the fully connected layer is the number of classes C, and the fully connected layer is followed by Softmax. The fully connected layer is used to perform high-dimensional feature mapping and class probability transformation on the temporal features output by the LSTM network to obtain the predicted probability distribution of sign language word category labels.

[0102] Initialize the parameters of the sign language vocabulary label prediction model, where ResNet18 is loaded with ImageNet pre-trained weights, and LSTM and fully connected layers are initialized using the Xavier method.

[0103] S2-2. Input each sample in the training set obtained in step S1 into the sign language vocabulary label prediction model. The sign language vocabulary label prediction model predicts the sign language vocabulary label and the sign language vocabulary label prediction probability distribution for each sample in the training set, as shown in the following formula:

[0104]

[0105] Where: i is the index of a sample in the training set; Predict the probability distribution of the sign language vocabulary category label for the i-th sample in the training set; For the i-th sample in the training set, after forward inference by the sign language vocabulary label prediction model, the probability value of being predicted as the c-th sign language vocabulary category is given.

[0106] Furthermore, after the sign language vocabulary label prediction model predicts the probability distribution of sign language vocabulary for each sample in the training set, the cross-entropy loss for each sample in the training set is calculated as the loss for that sample. The calculation formula is as follows:

[0107]

[0108] in This represents the true label encoding value of the i-th sample corresponding to the c-th sign language vocabulary category, i.e.:

[0109] The cross-entropy loss of the i-th sample in the training set is the loss of the i-th sample.

[0110] S2-3. Calculate the sampling probability of each sample in the training set based on the sample loss, as follows:

[0111] First, let the loss of all samples in the training set be... Where N is the number of samples in the training set. The loss for each sample in the training set is normalized to obtain...

[0112]

[0113] in: Let be the normalized loss for the i-th sample in the training set, and The corresponding samples are difficult samples.

[0114] Then, an exponential weighting method is used to calculate the sampling probability of each sample in the training set based on the normalized loss of each sample, in order to increase the probability of hard samples in the training set. The calculation is shown in the following formula:

[0115]

[0116] in ; Let be the sampling probability of the i-th sample based on the sample loss.

[0117] Finally, verify the probability distribution to ensure... And the training set has hard samples (i.e. Probability and percentage ≥ If the verification passes, the sampling probability is... It can be directly applied to subsequent key steps.

[0118] If the verification fails, adjust the temperature parameters. Recalculate the sampling probability And verify.

[0119] Step S3: Estimate the sampling probability of each sample in the training set for each time period, and obtain the sampling probability of each sample for each time period. The process is as follows:

[0120] S3-1. For each sample, i.e., the effective action frame sequence when each sign language performer performs each sign language word, let the duration of each sample be T. The sliding window method is used to divide the effective action frame sequence in each sample into multiple effective action frame stages. In the sliding window method, the window size... Frame, Step Frames. Therefore, the effective action frame stage set for each sample is obtained using the sliding window method. Where M is the number of stages in each sample, and ; For the Mth effective action frame stage in each sample For each sample, the m-th valid action frame stage contains W frames.

[0121] S3-2, Set the effective action frame stages for each sample The data is input into the sign language vocabulary label prediction model in step S2, and the effective action frame stage set for each sample is predicted by the sign language vocabulary label prediction model. In the sample, the sign language vocabulary category prediction label for each effective action frame stage, and the sign language vocabulary category label prediction probability distribution for each effective action frame stage of each sample.

[0122] Specifically, the ResNet18 network of the sign language vocabulary label prediction model obtains 512-dimensional intermediate layer features (i.e., single-frame RGB features) for each effective action frame stage of each sample. The average of these 512-dimensional intermediate layer features for each effective action frame stage of each sample is then calculated to obtain the stage feature vector for each effective action frame stage of each sample. That is, for the m-th effective action frame stage of each sample... The m-th effective action frame stage obtained Stage feature vector .

[0123] Then, the stage feature vector of each effective action frame stage of each sample is input into the LSTM network of the sign language vocabulary label prediction model. The LSTM network of the sign language vocabulary label prediction model captures temporal features, and the fully connected base layers of the sign language vocabulary label prediction model classify the temporal features to obtain the sign language vocabulary category prediction label for each effective action frame stage of each sample, as well as the predicted probability distribution of the sign language vocabulary category label for each effective action frame stage of each sample. in The probability distribution of the predicted sign language vocabulary category label for the m-th effective action frame stage for each sample. This represents the probability value of the c-th sign language vocabulary category label for the m-th valid action frame stage of each sample.

[0124] S3-3, the sign language vocabulary label prediction model predicts each effective action frame stage for each sample, and then uses cross-entropy loss to calculate the stage loss for each effective action frame stage of each sample. The calculation formula is as follows:

[0125]

[0126] in The true label encoding value for the c-th sign language vocabulary category corresponding to the m-th effective action frame stage; The stage loss for the m-th valid action frame stage for each sample.

[0127] Thus, we obtain the set of stage losses for all valid action frame stages for each sample. .in, The stage loss for the Mth effective action frame stage of each sample is... .

[0128] S3-4. Normalize the stage loss of each effective action frame stage for each sample to obtain the normalized stage loss of each effective action frame stage for each sample, as shown in the following formula:

[0129]

[0130] in: The original stage loss for the m-th valid action frame stage of each sample; The minimum original stage loss for all valid action frames in a single sample; The maximum value of the original stage loss for all valid action frames in a single sample; The normalized stage loss is the stage loss for the m-th effective action frame stage of each sample.

[0131] S3-5. Using an exponentially weighted method, normalized stage loss is calculated based on all valid action frame stages for each sample. Calculate the initial sampling probability for each sample in each effective action frame phase using the following formula:

[0132]

[0133] in: The initial sampling probability for the m-th valid action frame stage of a single sample.

[0134] S3-6. Normalize the initial sampling probability of each sample for each effective action frame stage again to obtain the final sampling probability of each sample for each effective action frame stage. As shown in the formula below:

[0135]

[0136] in: This represents the final sampling probability of the m-th effective action frame stage for a single sample. The initial sampling probability for the m-th effective action frame stage of a single sample; The initial sampling probability for the nth valid action frame stage of a single sample.

[0137] Step S3-7: Based on the final sampling probabilities of all valid action frame stages within each sample obtained in step S3-6, the sampling probability of the time stage of that sample is calculated using the maximum value aggregation strategy. The formula is as follows:

[0138]

[0139] in: The sampling probability of the i-th sample in a time period represents the criticality of that time period. The final sampling probability of the m-th stage of the i-th sample; K is the total number of effective action frame stages divided by the i-th sample (consistent with M in step S3-1).

[0140] This embodiment chooses maximum aggregation instead of summation / average aggregation because the core reason is that a high value of a sample only requires the existence of at least one key stage ( (The critical phases are not all critical phases.) Using summation and aggregation can obscure the fact that only a single phase is critical, leading to misjudgments in subsequent screening.

[0141] Step S4: Based on the sampling probability of each sample obtained in step S2, based on the sample loss. The sampling probability of each sample time stage obtained in step S3 is used to screen out high-value samples.

[0142] Uncertainty estimation is performed on each selected high-value sample to further filter out high-uncertainty samples, which are then used as anchor samples to construct a high-uncertainty sample set. Next, contrastive learning sample pairs are constructed for each anchor sample in the high-uncertainty sample set, resulting in augmented sample pairs for each anchor sample. Finally, the contrastive loss of each augmented sample pair is calculated, and the average contrastive loss of all augmented sample pairs is calculated, thereby estimating the contrastive loss of the high-uncertainty samples in the high-uncertainty sample set. The specific process is as follows:

[0143] S4-1. Compare the sampling probability of each sample based on the sample loss with a preset first probability threshold, and compare the sampling probability of each sample in a time period with a preset second probability threshold. When the sampling probability of a sample based on the sample loss is greater than or equal to the preset first probability threshold, and the sampling probability in a time period is greater than or equal to the preset second probability threshold, then the sampling probability of the sample meets the requirements, and the sample is a high-value sample with high loss sampling probability and high time period sampling probability.

[0144] S4-2. For each selected high-value sample, perform 5 forward inferences through the Dropout layer to obtain the predicted probability distribution of all sign language vocabulary category labels for each high-value sample. , These are the predicted probability distributions of the five sign language vocabulary category labels for the i-th high-value sample.

[0145] S4-3. Using the entropy method, the probability distribution of each high-value sample is predicted based on the five sign language vocabulary category labels of each sample, and the uncertainty of each high-value sample is calculated as shown in the following formula:

[0146]

[0147] in: The number of inferences; is the true label encoding value of the c-th sign language vocabulary category corresponding to the ith high-value sample during the k-th inference; Let represent the uncertainty of the i-th sample.

[0148] S4-4. Setting an uncertainty threshold In this embodiment .

[0149] The uncertainty of each high-value sample is compared with the uncertainty threshold. Compare these to filter out those with uncertainty greater than or equal to an uncertainty threshold. High-value samples are used as high-uncertainty samples, i.e., anchor samples. That is, when the uncertainty of the i-th high-value sample is... If the i-th high-value sample is a high-uncertainty sample, then the i-th high-value sample is the anchor sample. The anchor samples are then selected to construct the high-uncertainty sample set. .

[0150] S4-5. For high uncertainty sample sets Each anchor sample in the complete training set Find at least three positive samples with the same true label for the sign language vocabulary category, and at least ten negative samples with different true labels for the sign language vocabulary category. Use each anchor sample and its positive and negative samples to form a sample pair corresponding to each anchor sample. That is, for a high-uncertainty sample set... The i-th anchor sample Find the i-th anchor sample from the training set. At least three positive samples with the same real labels for the sign language vocabulary category At least ten negative samples whose sign language vocabulary categories differ from the actual labels. , with the i-th anchor sample The i-th anchor sample consists of positive samples of the same category and negative samples of different categories. The corresponding sample pairs.

[0151] Then, data augmentation is performed on the anchor sample, positive sample, and negative sample in each sample pair, and the augmented anchor sample, positive sample, and negative sample constitute the augmented sample pair corresponding to each sample. ,in For the enhanced i-th anchor sample, For the enhanced i-th anchor sample, It is the first negative sample of the i-th sample after enhancement.

[0152] S4-6. Input each enhanced sample pair into the ResNet18 network of the sign language vocabulary label prediction model in step S2, and extract a 512-dimensional feature vector through the ResNet18 network. ,in Let represent the 512-dimensional feature vector of the i-th anchor sample in the high uncertainty sample set, its corresponding enhanced positive sample 512-dimensional feature vector, and its corresponding enhanced negative sample 512-dimensional feature vector, respectively.

[0153] Then, the InfoNCE loss when the ResNet18 network extracts the 512-dimensional feature vector is calculated as the contrastive loss for each augmented sample pair. The calculation formula is as follows:

[0154]

[0155] in Represents the dot product of vectors; This represents the contrastive loss for each augmented sample pair.

[0156] Finally, the average contrastive loss of all enhanced sample pairs is calculated to obtain the high-uncertainty sample set. The contrastive loss for samples with medium to high uncertainty is shown in the following formula:

[0157]

[0158] in: Representing a sample set with high uncertainty The number of samples with medium to high uncertainty; Representing a sample set with high uncertainty The average InfoNCE loss of all augmented sample pairs corresponds to the i-th sample with high uncertainty. For high uncertainty sample sets The average contrast loss.

[0159] In this embodiment, the sampling probability of each sample obtained in step S2 based on the sample loss reflects the difficulty of classifying the sample, and the sampling probability of each sample time stage obtained in step S3 reflects the criticality of the sample time stage. Together, they screen out high-value samples that are "difficult to classify and have critical time stages". Then, through uncertainty calculation, the model predicts fuzzy high-uncertainty samples from the high-value samples to further accurately locate them, avoiding invalid calculations for simple samples and non-critical stage samples, and improving screening efficiency.

[0160] Steps S2 and S3 obtain the sampling probabilities, which serve as an initial screening tool to quickly narrow down the range of high-value samples based on sample loss and time characteristics. The uncertainty calculation (entropy method) in step S4 serves as a precise screening tool, which further determines whether a sample is an ambiguous sample that is difficult to distinguish from the model's true value based on the stability of the model's prediction (Dropout multi-round inference differences). The two steps form a progressive screening logic of "coarse screening + fine screening".

[0161] Step S4 Calculation The fundamental purpose is to transform high-uncertainty sample sets The feature optimization requirements of all high-uncertainty samples are integrated into a unified loss signal. If individual sample pairs are used directly... The total loss of all sample pairs may not accurately reflect the overall optimization needs of a high-uncertainty sample population; and By using a two-stage average—between the sample and the sample mean—the influence of the sample on quantitative differences is eliminated, while still ensuring representativeness. The overall feature differentiation level provides a stable and effective optimization direction for the total loss function.

[0162] Step S5: Construct a basic recognition model. The basic recognition model is trained using samples from the training set based on the calculation results of steps S2, S3, and S4. The trained basic recognition model is then validated using a validation set, thus obtaining the sign language recognition model. The specific explanation is as follows.

[0163] (1) In this embodiment, the constructed basic recognition model includes the calculation logic of the contrastive loss in step S4 to ensure that the contrastive loss can effectively affect model training. The basic recognition model includes a ResNet18 network (for spatial feature extraction), an LSTM network (for temporal feature capture), and a fully connected classification head. At the same time, a contrastive learning head is set to receive the feature vectors of high-uncertainty samples obtained in step S4 and calculate the InfoNCE contrastive loss. Furthermore, a temporal attention layer is set between ResNet18 and LSTM.

[0164] The time-attention layer weights features at different stages, as shown in the following formula:

[0165]

[0166] in: These are learnable parameters; Let be the attention weights for the m-th effective action frame stage; This is the stage feature vector of the m-th valid action frame stage of the sample; The attention score for the m-th valid action frame of the sample; The final stage fusion features are the samples after being weighted by a time attention mechanism.

[0167] Attention weights in the time attention layer Calculate the sampling probability of the time stage obtained in step S3, and then calculate the sampling probability of the sample time stage. For critical stages where the threshold value is ≥ a preset threshold (0.2 in this embodiment), a higher attention weight is initially assigned, such as... The initial value is set to 0.6; non-critical stage. The initial value is set to 0.1, and subsequent training iterations are used to optimize the model and ensure that it prioritizes the features of key stages.

[0168] During the initialization of the basic recognition model, the ResNet18 network is loaded with ImageNet pre-trained weights, and the temporal attention mechanism module, LSTM network, contrastive learning head, and fully connected classification head are all initialized using Xavier.

[0169] In this embodiment, the basic recognition model is designed with dual loss output, simultaneously outputting two loss branches, wherein:

[0170] Fully connected classification header outputs classification cross-entropy loss. This is used to optimize the model's ability to classify known categories.

[0171] The contrast learning head outputs a contrast loss. This contrast loss is directly reused from the average contrast loss calculated in step S4. Or based on the high uncertainty sample set in step S4 Real-time calculation of high-uncertainty samples ensures that losses are correlated with high-value samples.

[0172] (2) In this embodiment, when training the basic recognition model, training parameters and batches are constructed based on the sampling probabilities obtained in steps S2 and S3. The sampling probabilities obtained in steps S2 and S3 are used as the core basis to solve the problem of inefficient training of samples without discrimination.

[0173] Among them, the total loss function during training The contrast loss obtained in step S4 is explicitly included and calculated using the following formula:

[0174]

[0175] In the formula, The preset weights can be fine-tuned using the validation set. The goal is to allow the basic recognition model to learn the classification task while simultaneously optimizing the high-uncertainty sample set through comparative loss. The feature distribution of samples with medium to high uncertainty improves the distinguishability of similar sign language movements.

[0176] In the weighted sampling logic of the training batch, the construction of each batch (batch size = 32) strictly follows the weighting rule based on the sampling probability of sample loss × the sampling probability of the time stage. The specific process of the weighted sampling logic of the training batch is as follows:

[0177] Step 1, Sample Screening

[0178] Based on the sampling probability obtained in step S2, which is based on sample loss , select from training set (0.3 is a preset threshold to ensure that difficult-to-classify samples) samples are used to form a candidate sample pool, which contains 40%-50% of the total training set.

[0179] The second step is time-phase cropping.

[0180] Based on the time-stage sampling probability obtained in step S3 Each sample in the candidate pool is filtered, and only those samples are retained. For time-phase frame sequences with a value ≥0.2 (0.2 is a preset threshold to ensure critical phases), redundant frames are removed. For example, if the original sample contains 90 frames, after cropping, 30-40 key frames are retained.

[0181] The third step is to supplement the core samples.

[0182] Based on the high uncertainty sample set in step S4 In each batch, a high-uncertainty sample set is forcibly included. The number of high-uncertainty samples in the batch is ≥8 (accounting for 25% of the total batch), ensuring that the model focuses on learning high-uncertainty samples.

[0183] Step 4: Batch weight allocation

[0184] For the 32 samples that were finally selected, based on The product of the values ​​assigns weights to each sample. During training, the gradient is calculated based on these weights. Samples with higher weights contribute more to the update of model parameters.

[0185] In this embodiment, the AdamW optimizer is used for optimization during training. The initial learning rate of the AdamW optimizer is... Weight decay The learning rate decays to its original value every 20 epochs. The learning rate decay strategy of the AdamW optimizer is based on the average contrastive loss obtained in step S4. Changes: If in 5 consecutive epochs If the decrease is less than 0.001, it indicates that feature optimization for high-uncertainty samples has reached a bottleneck, and the learning rate should be reduced from... Attenuation to This ensures that the model continues to converge.

[0186] (3) Iterative logic of training and validation of basic recognition model

[0187] In this embodiment, the training direction is monitored through the calculation outputs of steps S2, S3, and S4 during the training process to prevent the basic recognition model from overfitting or deviating from the core optimization objective. A total of 100 epochs are trained, with each epoch containing... One batch. Used every 5 epochs. Evaluate and calculate the validation set accuracy. The specific iterative process is as follows:

[0188] During each epoch of training, the "key stage frame sequence" (rather than complete video frames) cropped in step S3 is received to reduce redundant computation. After the ResNet18 network extracts spatial features, the temporal attention layer is based on the sampling probability of the temporal stage. The initial weights (which are then optimized in subsequent iterations) focus on features at key stages, outputting weighted spatial-temporal fusion features. An LSTM network and a classification head are used to calculate the classification cross-entropy loss. Comparative learning head based Calculate the average contrast loss for the sample and according to the total loss function Backpropagation updates the model parameters.

[0189] Because changes in model parameters can lead to changes in sample loss and stage importance, the sampling based on sample loss in step S2 is recalculated every 10 epochs. And the sampling probability based on time stage in step S3 This ensures the timeliness of subsequent batch sampling.

[0190] Every 5 epochs, when evaluating with the validation set, the accuracy of a subset of high-uncertainty samples (selected from the validation set according to step S4 logic) is additionally calculated. If the accuracy of this subset of high-uncertainty samples increases by ≥1%, and the overall validation set accuracy increases by ≥0.5%, it indicates that the model's ability to identify difficult-to-classify samples has improved, and the current training is deemed effective.

[0191] In this embodiment, an early stopping strategy is used during training. The trigger condition for the early stopping strategy is: if the accuracy of the high uncertainty sample subset does not improve (or decreases) in 10 consecutive epochs, even if the overall validation set accuracy increases slightly, early stopping is triggered (to avoid the model only fitting simple samples and ignoring high-value samples).

[0192] Ultimately, during training in this embodiment, only the model weights with the highest accuracy in the high-uncertainty sample subset are saved (rather than just the overall accuracy), ensuring that the saved model weights have the best recognition ability for difficult-to-classify samples.

[0193] After the basic recognition model is trained, it is validated using a validation set, and the validation set accuracy is saved. Highest model weight The saved optimal model weights (Modelbest) fully contain the parameters of all learnable layers during the training of the basic recognition model, covering the entire process from feature extraction to loss calculation, and not just retaining parameters from a portion of the layers. Furthermore, the training loss is recorded after training is complete. Classification loss Comparative loss And the curves showing the accuracy of the training set / validation set.

[0194] Therefore, by training the basic recognition model, the model weights with the highest accuracy on the high-uncertainty sample subset and the validation set accuracy are obtained. The base recognition model with the highest model weight is the sign language recognition model.

[0195] Step S6: Use the test set obtained in step S1. The sign language recognition model obtained in step S5 is tested as follows:

[0196] C1. First, the test set Each sample in the dataset undergoes the same preprocessing as the training set (including background subtraction, cropping, and normalization). The training set samples (cropped RGB action frames) Before inputting into ResNet18, a channel mean subtraction operation (implicit normalization) matching the ResNet18 pre-trained weights is performed. Furthermore, it is ensured that the test set samples were not used in training / validation and that their distribution is essentially the same as the training set.

[0197] C2, will Sample input in ,get Predicted sign language labels for each sample Probability of prediction with sign language vocabulary labels .

[0198] Furthermore, the accuracy of the test set is calculated. ,in: This is an indicator function (1 if the condition is met, 0 otherwise); The true labels for the sign language vocabulary of the i-th sample; Predict the sign language vocabulary label for the i-th sample; For the test set test set The total number of samples.

[0199] Simultaneously, calculate the confusion matrix. ,in Indicates real label Predicted tags The number of samples. It is an index of sign language vocabulary categories, and it matches the true labels of the test samples. Predicted tags Direct association, i.e. , .

[0200] C3. Analysis of test results and generalization ability

[0201] Specifically, compare the accuracy rates on the training set, validation set, and test set. and The difference in accuracy If it is, then it is considered to be a good generalization.

[0202] For class pairs with high error rates in the confusion matrix, such as Larger values ​​for c1 and c2 are used to analyze the similarity of sign language movements (hand shape, movement trajectory), providing direction for dataset optimization.

[0203] Furthermore, the average inference time per sample of the model is recorded to evaluate its real-time performance. If the value is ms, then the real-time requirement is met.

[0204] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. These embodiments are merely descriptions of preferred embodiments and are not intended to limit the scope or concept of the invention. The specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. Such combinations, as long as they do not violate the spirit of the present invention, should also be considered as part of this disclosure. To avoid unnecessary repetition, the present invention will not further describe the various possible combinations.

[0205] This invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this invention and without departing from the design idea of ​​this invention, all modifications and improvements made by those skilled in the art to the technical solutions of this invention should fall within the protection scope of this invention. The technical content for which protection is sought in this invention has been fully described in the claims.

Claims

1. A method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning, characterized in that, The process is as follows: Step S1: Obtain sign language video data of multiple sign language performers performing multiple sign language words respectively, and determine the sign language action area from each RGB frame in each sign language video data, thereby obtaining each sign language action frame in the sign language video data of each sign language performer performing each sign language word. Then, effective action frames are selected from each sign language action frame in the sign language video data of each sign language performer performing each sign language word to construct an effective action frame sequence; Each valid action frame sequence is used as a sample, and a corresponding sign language vocabulary category real label is added to each sample. The dataset consists of each sample and the corresponding sign language vocabulary category real label, and the dataset is divided into training set, validation set and test set. Step S2: Input each sample in the training set obtained in Step S1 into a sign language vocabulary label prediction model to predict the sign language vocabulary category label. Estimate the sampling probability of each sample based on the loss of each sample to obtain the sampling probability of each sample based on the sample loss. The sign language vocabulary label prediction model includes a two-dimensional convolutional neural network (2D CNN), an LSTM network, and a fully connected layer, wherein the two-dimensional convolutional neural network (2D CNN) adopts a ResNet18 network. Step S3: Estimate the sampling probability of each sample in the training set for each time period to obtain the sampling probability of each sample in each time period. Step S4: Based on the sampling probability of each sample obtained in step S2, based on the sample loss. The sampling probability of each sample time stage obtained in step S3 is used to screen out high-value samples. Step S5: Construct a basic recognition model. The basic recognition model is trained using samples from the training set based on the calculation results of steps S2, S3, and S4. The trained basic recognition model is then validated using a validation set, thereby obtaining the sign language recognition model.

2. The method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning according to claim 1, characterized in that, In step S1, a background subtraction algorithm is used to remove the background from each RGB frame of the sign language video data of each sign language performer performing each sign language word, so as to obtain the foreground mask frame corresponding to each RGB frame of the sign language video data of each sign language word. Next, based on each foreground mask frame in the sign language video data for each sign language word, the minimum bounding box of the sign language action region in the corresponding RGB frame is calculated; Then, the initial minimum bounding box is optimized and corrected by combining the spatial information of the synchronous depth frame; Finally, the corresponding RGB frames and depth frames are cropped according to the optimized and corrected minimum bounding box of each sign language action region, thereby obtaining sign language action frames from each RGB frame in the sign language video data of each sign language performer performing each sign language word.

3. The method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning according to claim 1, characterized in that, In step S1, for each sign language action frame in the sign language video data of each sign language performer performing each sign language word, multiple key point landmarks of the hand in each sign language action frame are extracted; The motion amplitude of key point landmarks in each sign language motion frame is calculated based on multiple key point landmarks. Set a motion amplitude threshold, and filter sign language motion frames with a motion amplitude greater than or equal to the motion amplitude threshold as valid motion frames in the sign language video data of each sign language performer performing each sign language word. Therefore, multiple valid action frames are selected from each sign language action frame in the sign language video data of each sign language performer performing each sign language word, forming a sequence of valid action frames for each sign language performer performing each sign language word.

4. The method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning according to claim 1, characterized in that, In step S2, each sample in the training set obtained in step S1 is input into the sign language vocabulary label prediction model. The sign language vocabulary prediction label and the sign language vocabulary label prediction probability distribution of each sample in the training set are predicted by the sign language vocabulary label prediction model, and the loss of each sample in the training set is obtained. Then, the loss of each sample in the training set is normalized to obtain the normalized loss of each sample. Finally, the sampling probability of each sample in the training set is calculated based on the normalized loss of each sample in the training set using the exponential weighting method.

5. The method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning according to claim 1, characterized in that, In step S3, for each sample, the effective action frame sequence in each sample is divided into multiple effective action frame stages using the sliding window method; The effective action frame stage set of each sample is input into the sign language vocabulary label prediction model in step S2. The sign language vocabulary label prediction model predicts the sign language vocabulary category prediction label of each effective action frame stage in the effective action frame stage set of each sample, as well as the sign language vocabulary category label prediction probability distribution of each effective action frame stage of each sample, and calculates the stage loss of each effective action frame stage of each sample. Then, the stage loss of each effective action frame stage of each sample is normalized to obtain the normalized stage loss of each effective action frame stage of each sample. Next, the exponential weighting method is used to calculate the initial sampling probability of each sample for each effective action frame stage based on the normalized stage loss of all effective action frame stages for each sample. Then, the initial sampling probability of each sample for each effective action frame stage is normalized again to obtain the final sampling probability of each sample for each effective action frame stage. Finally, based on the final sampling probability of all valid action frame stages within each sample, the sampling probability of the time stage of that sample is calculated using a maximum value aggregation strategy.

6. The method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning according to claim 1, characterized in that, In step S4, the sampling probability of each sample based on the sample loss is first compared with a preset first probability threshold, and the sampling probability of each sample in a time period is compared with a preset second probability threshold. When the sampling probability of a sample based on the sample loss is greater than or equal to the preset first probability threshold, and the sampling probability in a time period is greater than or equal to the preset second probability threshold, then the sample is a high-value sample. Then, for each high-value sample, five forward inferences are performed through the Dropout layer to obtain the predicted probability distribution of all sign language word category labels for each high-value sample. Next, the entropy method is used to predict the probability distribution of all five sign language vocabulary category labels for each high-value sample, and the uncertainty of each high-value sample is calculated. Then, the uncertainty of each high-value sample is compared with the uncertainty threshold to select high-value samples whose uncertainty is greater than or equal to the uncertainty threshold, and these samples are used as high-uncertainty samples, i.e. anchor samples. For each anchor sample in the high uncertainty sample set, find at least three positive samples with the same true label of sign language vocabulary category and at least ten negative samples with different true labels of sign language vocabulary category in the complete training set. Each anchor sample and its positive and negative samples constitute the sample pair corresponding to each anchor sample, and enhance the positive and negative samples in each anchor sample and its sample pair. Then, each enhanced sample pair is input into the ResNet18 network of the sign language vocabulary label prediction model in step S2. The 512-dimensional feature vector is extracted by the ResNet18 network, and the InfoNCE loss when the sign language vocabulary label prediction model extracts the 512-dimensional feature vector is calculated as the contrast loss of each enhanced sample pair. Calculate the average contrastive loss of all enhanced sample pairs to obtain the contrastive loss of high-uncertainty samples in the high-uncertainty sample set.

7. The method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning according to claim 6, characterized in that, In step S5, the basic recognition model constructed includes a ResNet18 network, an LSTM network, and a fully connected classification head. At the same time, a contrastive learning head is set up to receive the feature vectors of high uncertainty samples obtained in step S4 and to calculate the InfoNCE contrastive loss. Furthermore, a temporal attention layer is set between the ResNet18 network and the LSTM. The attention weights of the temporal attention layer are calculated with reference to the time-stage sampling probabilities obtained in step S3. In the critical phase, higher attention weight is initially assigned; in non-critical phases... The initial value was set to 0.1, and subsequent optimization was achieved through training iterations. Fully connected classification header outputs classification cross-entropy loss. This is used to optimize the model's ability to classify known categories; The contrastive learning head outputs a contrastive loss, which can be directly reused from the average contrastive loss calculated in step S4, or based on the high-uncertainty sample set in step S4. Real-time calculation of high-uncertainty samples ensures that losses are correlated with high-value samples.

8. The method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning according to claim 7, characterized in that, In step S5, when the basic recognition model is trained, it first uses the sampling probability based on sample loss obtained in step S2. , select from training set Samples with a preset threshold are used to form a candidate sample pool; Next, based on the time-stage sampling probability obtained in step S3 Each sample in the candidate pool is filtered, and only those samples are retained. Redundant frames are removed from the time-series frames that meet a preset threshold. Subsequently, based on the high uncertainty sample set in step S4 In each batch, a high-uncertainty sample set is forcibly included. The number of high-uncertainty samples in the model is greater than or equal to a preset threshold to ensure that the model focuses on learning high-uncertainty samples. Finally, based on the selected samples, The product of these factors assigns weights to each sample.

9. The method for constructing a sign language recognition model based on uncertainty sample screening and comparative learning according to claim 7, characterized in that, Total loss function during training The contrast loss obtained in step S4 is incorporated and calculated using the following formula: In the formula, The preset weights are then optimized using the validation set. This represents the average contrast loss for a high-uncertainty sample set.

Images