Sign language recognition method and device based on cross-stage focus distillation and multi-branch temporal learning
By employing a sign language recognition method based on cross-stage focused distillation and multi-branch temporal learning, the problem of incomplete feature modeling in sign language recognition is solved, achieving efficient recognition of continuous sign language and improving recognition accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV OF TECH
- Filing Date
- 2026-04-16
- Publication Date
- 2026-07-24
AI Technical Summary
Existing sign language recognition technologies suffer from problems in feature modeling, such as incomplete semantics in shallow networks and loss of details in deep features. Traditional temporal networks struggle to balance short-term action units with long-term contextual dependencies.
A sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning is adopted. The method uses multi-branch temporal learning units to model short, medium and long-range action dependencies in parallel, and uses cross-stage focus distillation units to enhance key features. It also combines convolutional neural networks and bidirectional long short-term memory networks for feature alignment.
It improves the sign language recognition model's ability to understand and generalize complex spatiotemporal patterns in continuous sign language, effectively resolves the contradiction between incomplete semantics of shallow features and loss of fine-grained information in deep features, and improves recognition accuracy.
Smart Images
Figure CN122049994B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of sign language recognition technology, specifically relating to a sign language recognition method and apparatus based on cross-stage focus distillation and multi-branch temporal learning. Background Technology
[0002] Sign language is a visually-based mode of information expression. Its communication relies on the coordinated use of various visual elements, including hand gestures, hand shapes, facial expressions, and body postures, and has gradually developed into a relatively independent language system over time. Its unique grammatical structure and spatiotemporal dependence make it difficult for untrained hearing people to understand, thus creating communication barriers in many social spheres. To address this, sign language recognition technology aims to convert continuous gestures into text to facilitate barrier-free communication between the hearing impaired and mainstream society. To resolve these issues, Sign Language Recognition (SLR) technology analyzes and processes continuous visual signals in sign language videos, achieving automatic mapping from action sequences to text or semantic sequences, providing technical support for cross-group information interaction.
[0003] From a task perspective, video-based sign language recognition research typically encompasses two main scenarios: Isolated Sign Language Recognition (ISLR) and Continuous Sign Language Recognition (CSLR). ISLR primarily models single words or fixed actions, with short input video clips, clear annotation information, and a relatively simple overall recognition process. CSLR, on the other hand, targets sentence-level or segment-level sign language expressions, requiring the overall modeling of multiple continuous actions within longer video sequences. Its recognition results depend not only on local action features but are also significantly influenced by contextual temporal relationships. Early CSLR methods, limited by data scale, often employed handcrafted features and Hidden Markov Models for video segmentation and recognition. With the development of deep learning, Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), with their powerful feature learning and temporal modeling capabilities, have become the mainstream frameworks in this field. Current deep learning-based methods typically include two main modules: feature extraction and sequence alignment. Feature extraction often uses computationally efficient two-dimensional convolutional networks to extract discriminative visual features from videos. Sequence alignment generally uses bidirectional long short-term memory networks to model long-range contextual dependencies and utilizes connection-temporal classification (CTC) loss to achieve effective alignment of frame sequences and label sequences, replacing the traditional hidden Markov model alignment mechanism.
[0004] Despite the improvements made by the above methods, current continuous sign language recognition methods have obvious limitations in feature modeling: shallow networks can capture details but the semantics are incomplete, while deep features, although global, lose key spatiotemporal details; at the same time, traditional temporal networks are difficult to take into account both short-term action units and long-term contextual dependencies. Summary of the Invention
[0005] The purpose of this invention is to address the problems raised in the background art by proposing a sign language recognition method and apparatus based on cross-stage focus distillation and multi-branch temporal learning.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] This invention proposes a sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning, comprising:
[0008] Acquire the sign language video to be recognized and input it into the trained sign language recognition model to obtain the recognition result;
[0009] The sign language recognition model includes a first stage, a first embedding module, a second stage, a second embedding module, a third stage, a third embedding module, a fourth stage, and an auxiliary module connected in sequence.
[0010] Each of the embedded modules includes a multi-branch temporal learning unit and a cross-stage focused distillation unit connected in sequence;
[0011] The auxiliary module includes a convolutional layer, a bidirectional long short-term memory network, and a fully connected layer connected in sequence, and the fully connected layer outputs the recognition result of the sign language video to be recognized.
[0012] Preferably, the first stage, the second stage, the third stage, and the fourth stage are the four stages in the ResNet34 network.
[0013] Preferably, in the multi-branch temporal learning unit, the input of the multi-branch temporal learning unit is processed by a shape reshaping operation, then passes through three branches, and the outputs of the three branches are concatenated. The concatenated result is then processed by a convolutional layer and a shape restoration operation to obtain a third feature. The third feature is then weighted and residually connected with the input of the multi-branch temporal learning unit to obtain an enhanced feature. The enhanced feature is the output of the multi-branch temporal learning unit, which serves as the input of the corresponding cross-stage focal distillation unit.
[0014] Preferably, the three branches are convolutional layers with different dilation rates.
[0015] Preferably, each of the cross-stage focal distillation units includes a channel focal assembly and a spatial focal assembly connected in sequence;
[0016] In the channel focus component, the input of the cross-stage focus distillation unit passes through the first max pooling layer and the first average pooling layer to obtain the first feature and the second feature. The first feature and the second feature pass through the multilayer perceptron, and the first feature and the second feature after passing through the multilayer perceptron are added together and then passed through the activation layer to obtain the third feature. The third feature is multiplied by the input of the cross-stage focus distillation unit to obtain the output of the channel focus component, which is used as the input of the spatial focus component.
[0017] In the spatial focus component, the input of the spatial focus component passes through the second max pooling layer and the second average pooling layer to obtain the fourth feature and the fifth feature. The fourth feature and the fifth feature are concatenated and then passed through the convolutional layer and the activation layer in sequence to obtain the spatial weight map. The spatial weight map is then multiplied with the input of the cross-stage focus distillation unit to obtain the output of the cross-stage focus distillation unit.
[0018] Preferably, the loss function of the sign language recognition model The formula is expressed as follows:
[0019] ;
[0020] in, For the CTC loss of the sign language recognition model, For the first Stage and First Distillation losses during the stage, It is 3. These are the weighting coefficients.
[0021] Preferably, the distillation loss is the distillation loss between any two adjacent stages, and when calculating the distillation loss, the first stage is considered. The output of the first stage undergoes a convolution operation, and then passes through the second stage. Embedded module, obtain the sixth feature, and then... The output of the stage passes through the first The cross-stage focal distillation unit in the embedded module yields the seventh feature. The loss between the sixth and seventh features is calculated using the mean square error to obtain the distillation loss.
[0022] Preferably, the formula for calculating the CTC loss of the sign language recognition model is as follows:
[0023] ;
[0024] in, This is the output of a bidirectional long short-term memory network. The target sequence.
[0025] A sign language recognition device based on cross-stage focused distillation and multi-branch temporal learning includes a processor and a memory storing a number of computer instructions. When the computer instructions are executed by the processor, they implement the steps of the sign language recognition method based on cross-stage focused distillation and multi-branch temporal learning.
[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0027] This sign language recognition method and device based on cross-stage focus distillation and multi-branch temporal learning achieves complementary deep and shallow features through a cross-stage focus distillation unit, and simultaneously captures temporal dynamics of different granularities with the help of a multi-branch temporal learning unit, thereby improving the overall modeling ability of key semantics and action evolution in continuous sign language streams.
[0028] The cross-stage focused distillation unit, through selective enhancement of channel and spatial dimensions, guides the sign language recognition model to focus on key discriminative regions (such as hand shapes and movement trajectories) in sign language videos, effectively resolving the contradiction between incomplete semantics of shallow features and loss of fine-grained information in deep features. The multi-branch temporal learning unit, through a parallel multi-branch convolutional structure, simultaneously models lexical-level short-term actions and sentence-level long-range dependencies, overcoming the shortcomings of traditional temporal networks in capturing multi-granular temporal dynamics. The synergy of these two units significantly improves the sign language recognition model's understanding and generalization ability of complex spatiotemporal patterns in continuous sign language. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the sign language recognition model in the sign language recognition method and device based on cross-stage focused distillation and multi-branch temporal learning of the present invention.
[0030] Figure 2 This is a schematic diagram illustrating the origin of the loss function in the sign language recognition model of this invention;
[0031] Figure 3 This is a schematic diagram of the structure of each multi-branch timing learning unit in this invention;
[0032] Figure 4 This is a schematic diagram of the structure of the cross-stage focal distillation unit in this invention. Detailed Implementation
[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.
[0035] In one embodiment, such as Figures 1-4 As shown, a sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning is provided, including:
[0036] Step 1: Obtain the sign language video to be recognized;
[0037] Among them, the sign language videos consist of a series including The RGB image structure of a frame is denoted as . ,in, For the first RGB image of the frame for A matrix of dimensions (e.g., specifically a matrix of dimensions) The number of channels C is 3. and These correspond to the height and width of the RGB image, respectively.
[0038] Step 2: Input the acquired sign language video into the trained sign language recognition model to obtain the recognition result;
[0039] like Figures 1-2 As shown, the sign language recognition model includes a first stage, a first embedding module, a second stage, a second embedding module, a third stage, a third embedding module, a fourth stage, and an auxiliary module connected in sequence.
[0040] Step 2.1, the first stage, the second stage, the third stage and the fourth stage are the four stages in the ResNet34 network (i.e., stage1, stage2, stage3 and stage4). The detailed structure is consistent with the four stages in the existing ResNet34 network, and will not be described in detail again.
[0041] Each stage outputs a four-dimensional tensor, and the output of each stage is represented as follows: , Indicates the first The output of the stage for A matrix of dimension , and The first The channel, height, and width of each stage, such as the output of the first stage, are... The output of the second stage is The output of the third stage is The output of the fourth stage is .
[0042] Step 2.2: Each embedded module includes a multi-branch temporal learning unit and a cross-stage focused distillation unit connected in sequence;
[0043] Step 2.2.1, as follows Figure 3 As shown, in each multi-branch temporal learning unit, the input of the multi-branch temporal learning unit undergoes a shape reshaping operation (i.e., ... Transform into After that, it goes through three branches, and the outputs of the three branches are concatenated (concatenated along the channel dimension to obtain) The number of channels), and the splicing result sequentially passes through a convolutional layer (using 1×1 convolution to perform channel fusion and compression, reducing the number of channels from...). Compress back ) and shape restoration operation (i.e., restore to After that, the third feature is obtained. The third feature is then weighted and residually connected with the input of the multi-branch temporal learning unit to obtain the enhanced feature. The enhanced feature is the output of the multi-branch temporal learning unit, which is used as the input of the corresponding cross-stage focus distillation unit.
[0044] The three branches are convolutional layers with different dilation rates (in this embodiment, the three branches are composed of one-dimensional convolutional layers with dilation rates of 1, 2 and 3, respectively, and the size of the convolutional kernel is 3).
[0045] It should be noted that the core function of the multi-branch temporal learning unit is to realize multi-branch temporal modeling. By deploying grouped one-dimensional convolutions with different dilation rates in parallel, it simultaneously captures short, medium and long-range action dependencies: short-range branches perceive subtle changes between frames, medium-range branches model lexical-level gesture units, and long-range branches associate sentence-level semantic rhythms. This structure overcomes the limitation of a single temporal network in being unable to take into account the temporal dynamics of different granularities, and provides multi-stage temporal representation capabilities for continuous sign language sequences.
[0046] Step 2.2.2, as follows Figure 4 As shown, each cross-stage focus distillation unit includes a channel focus component and a spatial focus component connected in sequence; the cross-stage focus distillation unit is used to selectively enhance features by weighting and modulating feature information through the channel focus component and the spatial focus component to highlight key discriminative features related to continuous sign language recognition and suppress redundant information;
[0047] In the channel focus component, the input of the cross-stage focus distillation unit passes through the first max pooling layer and the first average pooling layer respectively to obtain the first feature ( ) and second feature ( The first and second features are processed by a multilayer perceptron, and the sum of the first and second features after processing through the multilayer perceptron is passed through an activation layer (Sigmoid activation) to obtain the third feature. The third feature is multiplied by the input of the cross-stage focal distillation unit (thereby enhancing the channel features containing key semantic information and suppressing redundant channel features, resulting in a feature map after channel focal component processing), and the output of the channel focal component is obtained. ), and serve as input for the spatial focus component;
[0048] In the spatial focus component, the input of the spatial focus component passes through the second max pooling layer and the second average pooling layer to obtain the fourth feature ( ) and the fifth feature ( The fourth and fifth features are concatenated and then passed through a convolutional layer (kernel size 7) and an activation layer (Sigmoid activation) to obtain a spatial weight map. The spatial weight map is then multiplied by the input of the cross-stage focal distillation unit (thus highlighting regions such as hands and faces that have high discriminative value for continuous sign language recognition in the spatial dimension, obtaining the feature representation after spatial focusing), to obtain the output of the cross-stage focal distillation unit. ).
[0049] Step 2.3: The auxiliary module includes a convolutional layer (the fourth convolutional layer in the ResNet34 network), a bidirectional long short-term memory network (BiLSTM), and a fully connected layer connected in sequence. The fully connected layer outputs the recognition result of the sign language video to be recognized.
[0050] In this embodiment, during the training phase of the sign language recognition model, the sign language recognition model is trained and forward inferenced using a training dataset (such as the weather forecast sign language dataset PHOENIX-2014, which contains 6841 sentences, of which 5672 sentences are used for training, 540 sentences are used for verification, and 629 sentences are used for testing. The PHOENIX-2014 dataset is expanded to obtain the PHOENIX14-t[4] dataset, which contains 1085 gestures (these 1085 gestures are used in the sentences of the PHOENIX-2014 dataset), of which 7096 gestures are used for training, 519 gestures are used for verification, and 642 gestures are used for testing). The loss function is calculated, and the parameters of the sign language recognition model are updated by backpropagation using the loss function until the sign language recognition model converges, thus obtaining the trained sign language recognition model. In this embodiment, the number of training rounds is 60, the batch size is 2, the optimizer is Adam, and the initial learning rate is 0.0001.
[0051] Loss function of sign language recognition model The formula is expressed as follows:
[0052] ;
[0053] in, For the CTC loss of the sign language recognition model, For the first Stage and First Distillation losses during the stage, It is 3. This is the weighting coefficient (e.g., take 1).
[0054] Distillation loss is the distillation loss between any two adjacent stages, and when calculating distillation loss, the first stage is considered. The outputs of the three stages (i.e., the first, second, and third stages respectively) are then processed by a convolution operation (this convolution operation makes the first stage...) The output of the first stage is comparable to that of the second stage in terms of channel count and spatial resolution. (Keep the output consistent across stages), then go through the next stage Embedded module, obtain the sixth feature, and then... The outputs of the three stages (i.e., the second, third, and fourth stages respectively) are processed by the... The cross-stage focused distillation unit in the embedded module yields the seventh feature. The loss between the sixth and seventh features is calculated using the mean squared error (MSE), resulting in the distillation loss (the distillation loss is used to evaluate the performance of the sixth feature). The learning process of the stage output is constrained, thereby enabling guided learning of shallow stage features by deep stage features, improving the sign language recognition model's ability to model key regions and multi-branch semantic information in continuous sign language videos, and Figure 2 The distillation loss in is ).
[0055] The formula for calculating the CTC loss of the sign language recognition model is as follows:
[0056] ;
[0057] in, This is the output of a bidirectional long short-term memory network. The target sequence.
[0058] To verify the importance of the multi-branch temporal learning unit and the cross-stage focus distillation unit in this method, validation and test data from the PHOENIX14 dataset (including the PHOENIX-2014 dataset and the PHOENIX14-t[4] dataset) were used to compare the word error rate of sign language recognition in four cases: without multi-branch temporal learning units and cross-stage focus distillation units, without multi-branch temporal learning units but with cross-stage focus distillation units, without cross-stage focus distillation units but with multi-branch temporal learning units, and with multi-branch temporal learning units and cross-stage focus distillation units. The results are shown in Table 1 (the checkmarks in Table 1 indicate the presence of the corresponding units).
[0059] Table 1
[0060]
[0061] As shown in Table 1, the word error rate of sign language recognition is the lowest when both multi-branch temporal learning units and cross-stage focus distillation units are present. Therefore, multi-branch temporal learning units and cross-stage focus distillation units significantly improve the accuracy of sign language recognition.
[0062] To verify the distillation loss of the first... The impact of different stages on sign language recognition accuracy was investigated using validation and test data from the PHOENIX14 dataset to compare word error rates in sign language recognition under different stage absence conditions, as shown in Table 2 (checkmarks in Table 2 indicate the presence of corresponding stages):
[0063] Table 2
[0064]
[0065] As shown in Table 2, among the distillation losses, the first... The stages are the second, third and fourth stages. When all three stages exist simultaneously, the word error rate of sign language recognition is the lowest. Therefore, the distillation loss corresponding to these three stages has a significant effect on improving the accuracy of sign language recognition.
[0066] To represent the weighting coefficients The impact of different weighting coefficient values on sign language recognition accuracy was investigated by comparing validation and test data from the PHOENIX14 dataset. The corresponding word error rates are shown in Table 3: Table 3
[0067]
[0068] According to Table 3, when the weighting coefficient When the value of is 1, the word error rate in sign language recognition is the lowest.
[0069] In another embodiment, the present invention also provides a sign language recognition device based on cross-stage focused distillation and multi-branch temporal learning, including a processor and a memory storing a number of computer instructions, which, when executed by the processor, implement the steps of the automatic response method that integrates dual word segmentation and iterative feedback.
[0070] For specific limitations on sign language recognition devices based on cross-stage focus distillation and multi-branch temporal learning, please refer to the limitations on sign language recognition methods based on cross-stage focus distillation and multi-branch temporal learning mentioned above, which will not be repeated here.
[0071] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, and the processor implements the method of the present invention by running the computer program stored in the memory.
[0072] The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store programs, which the processor executes upon receiving execution instructions.
[0073] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.
[0074] This sign language recognition method and device based on cross-stage focus distillation and multi-branch temporal learning achieves complementary deep and shallow features through a cross-stage focus distillation unit, and simultaneously captures temporal dynamics of different granularities with the help of a multi-branch temporal learning unit, thereby improving the overall modeling ability of key semantics and action evolution in continuous sign language streams.
[0075] The cross-stage focused distillation unit, through selective enhancement of channel and spatial dimensions, guides the sign language recognition model to focus on key discriminative regions (such as hand shapes and movement trajectories) in sign language videos, effectively resolving the contradiction between incomplete semantics of shallow features and loss of fine-grained information in deep features. The multi-branch temporal learning unit, through a parallel multi-branch convolutional structure, simultaneously models lexical-level short-term actions and sentence-level long-range dependencies, overcoming the shortcomings of traditional temporal networks in capturing multi-granular temporal dynamics. The synergy of these two units significantly improves the sign language recognition model's understanding and generalization ability of complex spatiotemporal patterns in continuous sign language.
[0076] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0078] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.
Claims
1. A sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning, characterized in that: The sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning includes: Acquire the sign language video to be recognized and input it into the trained sign language recognition model to obtain the recognition result; The sign language recognition model includes a first stage, a first embedding module, a second stage, a second embedding module, a third stage, a third embedding module, a fourth stage, and an auxiliary module connected in sequence. Each of the embedded modules includes a multi-branch temporal learning unit and a cross-stage focused distillation unit connected in sequence; The auxiliary module includes a convolutional layer, a bidirectional long short-term memory network, and a fully connected layer connected in sequence, and the fully connected layer outputs the recognition result of the sign language video to be recognized; The loss function of the sign language recognition model. The formula is expressed as follows: ； in, For the CTC loss of the sign language recognition model, For the first Stage and First Distillation losses during the stage, It is 3. These are the weighting coefficients; The distillation loss refers to the distillation loss between any two adjacent stages, and when calculating the distillation loss, the first stage is considered. The output of the first stage undergoes a convolution operation, and then passes through the second stage. Embedded module, obtain the sixth feature, and then... The output of the stage passes through the first The cross-stage focal distillation unit in the embedded module yields the seventh feature. The loss between the sixth and seventh features is calculated using the mean square error to obtain the distillation loss.
2. The sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning as described in claim 1, characterized in that: The first, second, third, and fourth stages are the four stages in the ResNet34 network.
3. The sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning as described in claim 1, characterized in that: In the multi-branch temporal learning unit, the input of the multi-branch temporal learning unit is processed by shape reshaping operation, then passes through three branches, and the outputs of the three branches are concatenated. The concatenated result is then processed by convolutional layer and shape restoration operation to obtain a third feature. The third feature is then weighted and residually connected with the input of the multi-branch temporal learning unit to obtain an enhanced feature. The enhanced feature is the output of the multi-branch temporal learning unit, which serves as the input of the corresponding cross-stage focal distillation unit.
4. The sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning as described in claim 3, characterized in that: The three branches are convolutional layers with different dilation rates.
5. The sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning as described in claim 1, characterized in that: Each of the aforementioned cross-stage focal distillation units includes a channel focal assembly and a spatial focal assembly connected in sequence; In the channel focus component, the input of the cross-stage focus distillation unit passes through the first max pooling layer and the first average pooling layer to obtain the first feature and the second feature. The first feature and the second feature pass through the multilayer perceptron, and the first feature and the second feature after passing through the multilayer perceptron are added together and then passed through the activation layer to obtain the third feature. The third feature is multiplied by the input of the cross-stage focus distillation unit to obtain the output of the channel focus component, which is used as the input of the spatial focus component. In the spatial focus component, the input of the spatial focus component passes through the second max pooling layer and the second average pooling layer to obtain the fourth feature and the fifth feature. The fourth feature and the fifth feature are concatenated and then passed through the convolutional layer and the activation layer in sequence to obtain the spatial weight map. The spatial weight map is then multiplied with the input of the cross-stage focus distillation unit to obtain the output of the cross-stage focus distillation unit.
6. The sign language recognition method based on cross-stage focus distillation and multi-branch temporal learning as described in claim 1, characterized in that: The formula for calculating the CTC loss of the sign language recognition model is as follows: ； in, This is the output of a bidirectional long short-term memory network. The target sequence.
7. A sign language recognition device based on cross-stage focused distillation and multi-branch temporal learning, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 6.