Continuous sign language recognition method and device based on sign language word prior guidance

Abstract

The embodiment of the application provides a continuous sign language recognition method and device based on sign language word prior guidance, relates to the technical field of continuous sign language recognition, and specifically comprises the following steps: acquiring a continuous sign language recognition data set, fine-tuning the continuous sign language recognition data set by using a pre-trained BERT model to extract sign language word features with action invariance, and constructing a visual feature extraction model to extract sign language visual features; constructing a cross-modal feature alignment target function and a CTC target function to construct a target function, and training the visual feature extraction model to form a cross-modal feature alignment model. Through the method and device provided by the application, the technical problem of weak generalization ability of visual features in continuous sign language recognition in the prior art can be alleviated, the sign language word features with action invariance are used as priori, the multiple video frame visual features corresponding to the sign language word features are constrained, and finally the difference between the visual features is reduced.

Description

Technical Field

[0001] This invention relates to the technical field of continuous sign language recognition, and in particular to a method and apparatus for continuous sign language recognition based on prior guidance of sign language words. Background Technology

[0002] As a means of communication in daily life for people with hearing impairments, sign language primarily transmits information through a combination of various information, including changes in gestures, mouth movements, facial expressions, and head and body movements. Continuous sign language recognition (CSLR), as a research project to promote barrier-free communication between people with hearing impairments and those with hearing, uses computer vision and natural language processing technologies to identify each sign language word in a sign language video while preserving its order.

[0003] Existing technologies employ the following methods: 1) Hand-designing visual features based on Gaussian mixture model (GMM)-hidden markov model (HMM). Compared to deep learning-based visual features, hand-designed visual features lack generalization ability on large datasets and cannot be optimized end-to-end;

[0004] 2) By combining deep learning models with HMM, a discriminative model for predicting frame-level labels in videos was established. However, the HMM-based method needs to utilize prior knowledge from the previous stage to optimize the model in the current stage.

[0005] 3) CTC is used to maximize the sum of probabilities of all feasible alignment paths between video frames and sign language words in sentences to obtain frame-level labels. Although CTC has achieved advanced performance and can optimize the model end-to-end, using CTC alone also faces the problem of weak generalization ability of visual features.

[0006] In summary, existing technologies suffer from the technical problem of weak generalization ability of visual features in continuous sign language recognition. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a continuous sign language recognition method and apparatus based on prior guidance of sign language words, so as to alleviate the technical problem of weak generalization ability of visual features in continuous sign language recognition in the prior art. The invention uses sign language word features with action invariance as priors to constrain the visual features of multiple corresponding video frames, thereby reducing the differences between visual features.

[0008] In a first aspect, embodiments of the present invention provide a continuous sign language recognition method based on prior guidance of sign language words, specifically including the following steps:

[0009] A continuous sign language recognition dataset is obtained, and a pre-trained BERT model is used to fine-tune the continuous sign language recognition dataset to extract sign language word features with action invariance. A visual feature extraction model is constructed to extract sign language visual features. The continuous sign language recognition dataset includes continuous sign language recognition videos and corresponding sign language word sequences.

[0010] A cross-modal feature alignment objective function and a CTC objective function are constructed to form an objective function. The objective function is then added to a pre-trained BERT model and a visual feature extraction model to form a cross-modal feature alignment model, and the cross-modal feature alignment model is trained.

[0011] Preferably, the visual feature extraction model includes ResNet18, a temporal feature extraction network based on parallel-dense connections, a BGRU model, and a fully connected layer;

[0012] The ResNet18 consists of four residual modules and one global average pooling layer.

[0013] Preferably, the modal feature alignment objective function includes:

[0014] Construct the cost function:

[0015] —It possesses the characteristics of sign language words with motion invariance;

[0016] —Visual features of sign language;

[0017] L cma —Cost function;

[0018]

[0019]

[0020] In the formula, P = K⊙M;

[0021]

[0022] μ—f Gls Discrete category probability distribution;

[0023] ν—f Gru Discrete category probability distribution;

[0024] β—a hyperparameter controlling the entropy contribution;

[0025] M (t+1) = diag(v)Pdiag(μ);

[0026]

[0027] Preferably, the steps of the CTC objective function include calculating the conditional probability of the alignment path π as follows:

[0028]

[0029] π—the set of alignment paths between all frames X in the video and their corresponding words;

[0030] C—Number of all word categories in the continuous sign language dataset;

[0031] blank—empty class;

[0032] Given X, the sum of probabilities of all feasible alignment paths is obtained using the following formula.

[0033]

[0034] B—The mapping that removes duplicate labels and blank classes from π;

[0035] The objective function for CTC optimization is:

[0036] L CTC = -log p(Y|X).

[0037] Preferably, the steps for constructing the objective function, including the cross-modal feature alignment objective function and the CTC objective function, include:

[0038] The objective function is:

[0039]

[0040] λ — a hyperparameter controlling the contribution of entropy.

[0041] θ s —Parameters of the ResNet18 model;

[0042] θ pdc —Parameters of a temporal feature extraction network based on parallel-dense connections and the BGRU model;

[0043] θ sq —BGRU model and fully connected layer parameters.

[0044] Preferably, the step of acquiring a continuous sign language recognition dataset and fine-tuning the dataset using a pre-trained BERT model to extract sign language word features with action invariance includes:

[0045] Construct a BERT model and pre-train it to obtain the sign language word sequence Y = {y1,...,y}. L The pre-trained BERT model was fine-tuned using a language model training strategy that masked the language model training strategy, and the continuous sign language recognition dataset was then used with sign language word sequences.

[0046] The BERT model consists of twelve stacked Transformer modules;

[0047]

[0048]

[0049]

[0050] These are the input and output features of the Tranformer module, respectively.

[0051] Norm—Normalization layer;

[0052] FF—Forward Propagation Layer.

[0053] Emb T —Word embedding layer;

[0054] Emb p (i) — Position encoder;

[0055] Attention—Attention mechanism layer.

[0056] Preferably, the specific structure of each residual module of the ResNet18 is as follows:

[0057]

[0058]

[0059] Norm—Batch Normalization Layer (BatchNorm);

[0060] ReLU — ReLU activation function;

[0061] Conv—a two-dimensional convolutional layer with a 3×3 kernel;

[0062]

[0063] Spatial information features of video V The output representation of the global average pooling layer.

[0064] Preferably, the temporal feature extraction network based on parallel-dense connections comprises m densely connected sub-parallel stacked blocks;

[0065] The operation of the l-th layer RDC in the m-th SRC block is as follows:

[0066]

[0067] and —Input and output features of the l-th layer RDC in the m-th SRC block.

[0068] * — Convolution operation;

[0069] —Refers to the weights and biases of the dilated convolutional layer.

[0070] d refers to the feature dimension;

[0071] Reusing the features of layer l of RDC in m SRC blocks:

[0072]

[0073] Preferably, after the step of training the sensory feature extraction model to form a cross-modal feature alignment model, the method further includes:

[0074] The cross-modal feature alignment model was tested by using CTC Beamsearch to predict sign language sentences from the model's output visual features, and then evaluating the predicted sentences using the WER metric, as shown in the following formula:

[0075]

[0076] n I ,n D ,n S —Number of insert, delete, and replace operations

[0077] L—The number of words in the standard reference sentence.

[0078] On the other hand, the present invention provides a continuous sign language recognition device based on prior guidance of sign language words, comprising:

[0079] Feature extraction module: used to acquire continuous sign language recognition dataset, fine-tune the continuous sign language recognition dataset using a pre-trained BERT model to extract sign language word features with action invariance, and construct a visual feature extraction model to extract sign language visual features. The continuous sign language recognition dataset includes continuous sign language recognition videos and corresponding sign language word sequences.

[0080] Cross-modal feature alignment model module: Construct cross-modal feature alignment objective function and CTC objective function to form objective function, add the objective function to the pre-trained BERT model and visual feature extraction model to form cross-modal feature alignment model, and train the cross-modal feature alignment model.

[0081] The embodiments of this invention bring the following beneficial effects: This invention provides a method and apparatus for continuous sign language recognition based on prior guidance of sign language words. The method specifically includes the following steps: acquiring a continuous sign language recognition dataset; fine-tuning the continuous sign language recognition dataset using a pre-trained BERT model to extract sign language word features with action invariance; constructing a visual feature extraction model to extract sign language visual features; the continuous sign language recognition dataset includes continuous sign language recognition videos and corresponding sign language word sequences; constructing a cross-modal feature alignment objective function and a CTC objective function to form the objective function; adding the objective function to the pre-trained BERT model and the visual feature extraction model to form a cross-modal feature alignment model; and training the cross-modal feature alignment model. The method and apparatus provided by this invention can alleviate the technical problem of weak generalization ability of visual features in continuous sign language recognition in the prior art. By using sign language word features with action invariance as priors, it constrains the visual features of multiple corresponding video frames, ultimately reducing the differences between visual features.

[0082] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0083] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0084] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0085] Figure 1 A flowchart of a continuous sign language recognition method based on prior guidance of sign language words is provided in an embodiment of the present invention;

[0086] Figure 2This is a schematic diagram illustrating the process of extracting action-invariant sign language word features from a continuous sign language recognition method based on prior guidance of sign language words, as provided in an embodiment of the present invention.

[0087] Figure 3 This is a schematic diagram illustrating the process of extracting visual features of a continuous sign language recognition method based on prior guidance of sign language words, provided in an embodiment of the present invention.

[0088] Figure 4 This is a diagram of a temporal feature extraction network structure based on parallel-dense connections, provided for an embodiment of the present invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0090] Currently, sign language, as a means of communication in daily life for the hearing impaired, primarily transmits information through a combination of various information, including gesture changes, mouth movements, facial expressions, and head and body movements. Continuous sign language recognition (CSLR), as a research method to promote barrier-free communication between the hearing impaired and hearing individuals, utilizes computer vision and natural language processing technologies to identify each sign language word in a sign language video while preserving its order. Based on this, this invention provides a continuous sign language recognition method based on prior guidance of sign language words. This method can alleviate the technical problem of weak generalization ability of visual features in existing continuous sign language recognition technologies. Furthermore, by using sign language word features with motion invariance as priors, it constrains the visual features of their corresponding multiple video frames, ultimately reducing the differences between visual features.

[0091] To facilitate understanding of this embodiment, a detailed description of a continuous sign language recognition method based on prior guidance of sign language words, as disclosed in this embodiment of the invention, will be provided first.

[0092] Example 1:

[0093] Combination Figure 1 Embodiment 1 of the present invention provides a continuous sign language recognition method based on prior guidance of sign language words, which specifically includes the following steps:

[0094] A continuous sign language recognition dataset is obtained, and a pre-trained BERT model is used to fine-tune the continuous sign language recognition dataset to extract sign language word features with action invariance. A visual feature extraction model is constructed to extract sign language visual features. The continuous sign language recognition dataset includes continuous sign language recognition videos and corresponding sign language word sequences.

[0095] A cross-modal feature alignment objective function and a CTC objective function are constructed to form an objective function. This objective function is then added to a pre-trained BERT model and a visual feature extraction model to form a cross-modal feature alignment model, which is then trained.

[0096] Example 2:

[0097] Combination Figure 3 and Figure 4 Embodiment 2 of the present invention elaborates on the aforementioned visual feature extraction model, including ResNet18, a temporal feature extraction network based on parallel-dense connections, the BGRU model, and a fully connected layer;

[0098] Specifically, the ResNet18 consists of four residual modules and one global average pooling layer.

[0099] It should be noted that, in the embodiments provided by the present invention, a short video V containing T frames is defined as V = {v1,...,v...} T Meanwhile, the ResNet18 pre-trained on the ImageNet dataset is defined as G. s (.;θ s The specific structure of each residual module in the ResNet18 is as follows:

[0100]

[0101]

[0102] Norm—Batch normalization layer;

[0103] ReLU — ReLU activation function;

[0104] Conv—a two-dimensional convolutional layer with a 3×3 kernel;

[0105]

[0106] Spatial information features of video V The output representation of the global average pooling layer.

[0107] Based on the acquisition of spatial information from sign language videos, and in order to capture the diverse sign language actions corresponding to different sign language words and alleviate the temporal differences in the demonstration of different sign language actions, this method designs a temporal feature extraction network (PDC) based on parallel-dense connections. pdc (.;θ pdc A PDC contains m sub-parallel stacked densely-connected blocks (SDCs). Each SDC also has l layers of residual dilated convolution (RDCs), which consist of 1D dilated convolutions, batch normalization (BN) layers, and ReLU activation functions.

[0108] The operation of the l-th layer RDC in the m-th SRC block is as follows:

[0109]

[0110] and —Input and output features of the l-th layer RDC in the m-th SRC block.

[0111] * — Convolution operation;

[0112] —Refers to the weights and biases of the dilated convolutional layer.

[0113] d refers to the feature dimension;

[0114] Furthermore, the expansion factor for all RDC layers within the same SDC block is 2. m-1 This means that m SDC blocks can be used to construct different temporal receptive fields, extracting visual features at different temporal resolutions, enabling the model to effectively recognize the sign language actions corresponding to different sign language words. To reuse information at different temporal resolutions and enhance the extraction of visual features at multiple temporal resolutions, this application employs dense connections, reusing the features of layer l of RDCs in the m SRC blocks. The specific operation is as follows:

[0115]

[0116] By employing this information reuse strategy, the model can acquire high-temporal-resolution information from the video while also considering low-temporal-resolution information, thus more effectively combining the two types of temporal-resolution information. Finally, the output features of all SDC blocks are fused to obtain the final video visual feature representation.

[0117] Furthermore, to obtain the sequence information of the entire sign language video, BGRU is used to model the correlation of all video frames. The input of this layer is the output feature f of PDC. MT Subsequently, the output features of the bidirectional gated recurrent unit (BGRU) layer are... The input is fed into a fully connected layer, producing a class probability matrix of class C. The combination of BGRU and fully connected layers is defined as G. sq (.;θ sq ).

[0118] Example 3:

[0119] Embodiment 3 of this invention describes the modal feature alignment target algorithm. Given a sign language video and the corresponding sign language word sequence, the BERT model extracts sign language word sequence features with action invariance. Visual features extracted from sign language videos using BGRU At the same time, given a cost function Obtain its cost matrix

[0120] Assume f Gls and f Gru Let the discrete class probability distributions be μ and v, and let... Calculate the transformation matrix from visual feature distribution to text feature distribution. and combine it with the cost matrix C ij Calculate the Frobenius dot product together to measure f. Gls and f Gru Alignment relationship:

[0121]

[0122] Where ∑jM ij =μ i ,∑iM ij =ν j , Finally, this method uses the Sinkhorn algorithm to solve for D(μ,ν), and obtains: Where H(M) = ∑ i,j M ij log M ij β is a hyperparameter that controls the contribution of the entropy part in the above formula;

[0123] Optimized with the Sinkhorn algorithm, this cross-modal feature alignment method can be deployed in current deep learning frameworks such as PyTorch, and the process of transforming visual feature distribution into text feature distribution is f Gls =M(f Gru ).

[0124] In conjunction with this application, the modal feature alignment objective function includes:

[0125] Construct the cost function:

[0126] —It possesses the characteristics of sign language words with motion invariance;

[0127] —Visual features of sign language;

[0128] L cma —Cost function;

[0129]

[0130]

[0131] In the formula, P = K⊙M;

[0132]

[0133] μ—f Gls Discrete category probability distribution;

[0134] v—f Gru Discrete category probability distribution;

[0135] β—a hyperparameter controlling the entropy contribution;

[0136] M (t+1) = diag(v)Pdiag(μ);

[0137]

[0138] Furthermore, in the embodiments provided by this invention, the optimal transmission distance is obtained using the following algorithm:

[0139] initialization M (1) =11 T ;

[0140] Define the formula:

[0141]

[0142] Iterative update: Let

[0143] Then update M (t+1) =diag(ν)Pdiag(μ).

[0144] In this application, the method here uses the connectionist temporal classification (CTC) algorithm to predict the video frame corresponding to the sign language word;

[0145] Preferably, the steps of the CTC objective function include calculating the conditional probability of the alignment path π as follows:

[0146]

[0147] π—the set of alignment paths between all frames X in the video and their corresponding words;

[0148] C—Number of all word categories in the continuous sign language dataset;

[0149] blank—empty class;

[0150] Given X, the sum of probabilities of all feasible alignment paths is obtained using the following formula.

[0151]

[0152] B—The mapping that removes duplicate labels and blank classes from π;

[0153] The objective function for CTC optimization is:

[0154] L CTC = -log p(Y|X).

[0155] Preferably, the steps for constructing the objective function, including the cross-modal feature alignment objective function and the CTC objective function, include:

[0156] The objective function is:

[0157]

[0158] λ — a hyperparameter controlling the contribution of entropy.

[0159] θ s —Parameters of the ResNet18 model;

[0160] θ pdc —Parameters of a temporal feature extraction network based on parallel-dense connections and the BGRU model;

[0161] θ sq —BGRU model and fully connected layer parameters.

[0162] This model uses the backpropagation algorithm to calculate the gradient of the model parameters and backpropagates the gradient according to the learning rate γ to update the model parameters until the model converges or reaches the required number of training iterations. The specific calculation process is as follows:

[0163]

[0164]

[0165]

[0166] Example 4:

[0167] Combination Figure 2 Embodiment 4 of this invention elaborates on the BERT model. Because different sign language demonstrators often exhibit visual differences when demonstrating the same sign language word due to personal habits, their visual feature representations differ to varying degrees. However, the features of the demonstrated sign language word do not possess visual or textual differences; they possess natural semantic invariance and can serve as a unified representation of all actions corresponding to that sign language word. This is referred to here as action invariance. Inspired by this, unlike most sign language recognition methods that map sign language words to one-hot encodings, this application uses a BERT model pre-trained with natural language to fine-tune a continuous sign language recognition dataset to extract textual features of sign language words.

[0168] Furthermore, given a sequence of sign language words containing L words, this sequence is defined as Y = {y1,...,y...} L By employing a Mask Language Modeling (MLM) training strategy, 50% of the sign language words in the sign language sentences are randomly masked during training.

[0169] Each sign language word y i Input to a word embedding layer Emb T , obtain y i The high-dimensional feature representation of Y is then processed by the position encoder Emb. P After being assigned positional encoding, the input is fed into 12 Transformer modules, and the output is a high-dimensional feature representation of the sign language word sequence Y. in The feature representation of each sign language word in the sequence. Refers to the feature dimension

[0170] The BERT model consists of twelve stacked Transformer modules;

[0171]

[0172]

[0173]

[0174] These are the input and output features of the Tranformer module, respectively.

[0175] Norm—Normalization layer;

[0176] FF—Forward Propagation Layer;

[0177] Emb T —Word embedding layer;

[0178] Emb p (i) — Position encoder;

[0179] Attention—Attention mechanism layer.

[0180] The goal of optimizing the BERT model is to successfully predict masked sign language words. Since the set of sign language words is a subset of the natural language word set, this MLM training strategy can maximize the discriminative power of the high-dimensional text features of different sign language words and obtain a certain level of sign language grammatical information. During training, the fine-tuned parameters of the BERT model are frozen, ensuring that its output sign language word sequence features possess action invariance, so that they can be further aligned with the visual features of video frames in high-dimensional space.

[0181] Example 5:

[0182] In the embodiments provided by this invention, the cross-modal feature alignment model is trained by first performing data augmentation on the samples, then redefining the size of all video frames, performing random multi-scale cropping, and randomly horizontally flipping. In the resizing, the video frames are redefined to (width * height) 256×256. Multi-scale cropping involves randomly selecting a cropping ratio from (1.0, 0.875, 0.75, 0.66) based on the redefined size, multiplying it by the target cropping size (224×224) to generate the cropping region, and finally redefining the video frame size to 224×224. Random horizontal flipping involves rotating all video frames horizontally by 180 degrees with a 50% probability. During testing, this invention only redefines the size (width * height) of the video frame to 256×256 and performs center cropping of the cropping area to (224×224). Center cropping is based on the redefinition size (width * height) of 256×256, and calculates and crops the center area of ​​the video frame according to the final video frame size (224×224).

[0183] To verify the effectiveness of this invention, it was evaluated on the continuous sign language recognition datasets RWTH-2014, RWTH-2014T, and CSL500. This invention uses CTC Beamsearch to predict the sign language sentences corresponding to sign language videos and employs the word error rate (WER) as an evaluation metric, which is widely used in continuous sign language recognition tasks [5,6,7,12,13]. WER measures the minimum number of "insertion," "deletion," and "replacement" operations required to convert a predicted sentence into a standard reference sentence; a lower WER indicates better evaluation performance.

[0184]

[0185] Where, n I ,n D ,n S These represent the number of "insert", "delete", and "replace" operations, respectively, and L is the number of words in the standard reference sentence.

[0186] To effectively mitigate video frame redundancy during training, similar to the frame selection strategy of SFL, this invention uniformly and randomly extracts half of all video frames for each video during training on the RWTH-2014 and RWTH-2014T datasets. For the CSL500 dataset, 20 frames are uniformly and randomly extracted from all video segments during training. For these three datasets, the experiment consisted of 80 training epochs. The model used the Adam optimizer with an initial learning rate of 1e–4. The learning rate was multiplied by 0.1 at the 30th and 60th epochs. Hyperparameters β and λ were set to 0.5 and 2, respectively, and the training batch size was set to 4. During model testing, the beam width of CTC Beamsearch was set to 10.

[0187] The performance comparison of other advanced continuous sign language recognition algorithms with the present invention is shown in Tables 1, 2 and 3. The ablation experiment of the cross-modal feature alignment model on the RWTH-2014 dataset is shown in Table 4.

[0188] Table 1 compares the results with other continuous sign language recognition algorithms on the RWTH-2014 dataset.

[0189]

[0190] Table 2 compares the results with other continuous sign language recognition algorithms on the RWTH-2014T dataset.

[0191]

[0192] Table 3 compares the results with other continuous sign language recognition algorithms on the CSL500 dataset.

[0193]

[0194]

[0195] Table 4 Ablation experiments of the cross-modal feature alignment model on the RWTH-2014 dataset.

[0196]

[0197] As shown in Tables 1, 2, and 3, the cross-modal feature alignment model proposed in this invention exhibits advanced recognition performance on multiple publicly available continuous sign language recognition datasets. It has been demonstrated that the parallel-densely connected temporal feature extraction network (PDC) proposed in this invention can effectively capture the diverse sign language actions corresponding to different sign language words in a sign language video. Furthermore, the cross-modal feature alignment algorithm proposed in this invention can effectively align the visual features of video frames with their corresponding sign language word features in a high-dimensional feature space, effectively improving the model's generalization ability.

[0198] Example 6:

[0199] This invention provides a continuous sign language recognition device based on prior guidance of sign language words, comprising:

[0200] A continuous sign language recognition dataset is obtained, and a pre-trained BERT model is used to fine-tune the continuous sign language recognition dataset to extract sign language word features with action invariance. A visual feature extraction model is constructed to extract sign language visual features. The continuous sign language recognition dataset includes continuous sign language recognition videos and corresponding sign language word sequences.

[0201] A cross-modal feature alignment objective function and a CTC objective function are constructed to form an objective function. This objective function is then added to a pre-trained BERT model and a visual feature extraction model to form a cross-modal feature alignment model, which is then trained.

[0202] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0203] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0204] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0206] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0207] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0208] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0209] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0210] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0211] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0212] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A continuous sign language recognition method based on prior guidance of sign language words, characterized in that, Specifically, the steps include the following: A continuous sign language recognition dataset is obtained, and a pre-trained BERT model is used to fine-tune the continuous sign language recognition dataset to extract sign language word features with action invariance. A visual feature extraction model is constructed to extract sign language visual features. The continuous sign language recognition dataset includes continuous sign language recognition videos and corresponding sign language word sequences. A cross-modal feature alignment objective function and a CTC objective function are constructed to form an objective function. The objective function is then added to a pre-trained BERT model and a visual feature extraction model to form a cross-modal feature alignment model, and the cross-modal feature alignment model is trained. The visual feature extraction model includes ResNet18, a parallel-dense connection-based temporal feature extraction network, a BGRU model, and a fully connected layer. The ResNet18 consists of 4 residual modules and 1 global average pooling layer; The modality feature alignment objective function includes: Construct the cost function: ; —It possesses the characteristics of sign language words with motion invariance; —Visual features of sign language; —Cost function; ； ， ， ； In the formula, ; ； ； — Discrete category probability distribution; — Discrete category probability distribution; —A hyperparameter that controls the contribution of entropy; = ； ； The parallel-dense connection-based temporal feature extraction network includes Densely connected blocks stacked in parallel; No. The first SRC block in The operations of the layer RDC are as follows: ， and —No. The first SRC block in Input and output characteristics of layered RDC — Convolution operation; , —Refers to the weights and biases of the dilated convolutional layer. Refers to the feature dimension; Reuse In each SRC block Characteristics of layered RDCs: 。 2. The method according to claim 1, characterized in that, The objective function of CTC is: Alignment path The conditional probability is calculated as follows: ； —All frames in the video The set of alignment paths for its corresponding words; —Number of all word categories in the continuous sign language recognition dataset; —Empty class; Given Under the given conditions, the sum of probabilities of all feasible alignment paths is obtained using the following formula: ； -Will Duplicate tags and The mapping removed from the class; The objective function for CTC optimization is: 。 3. The method according to claim 2, characterized in that, The steps for constructing the cross-modal feature alignment objective function and the CTC objective function include: The objective function is: ； —A hyperparameter that controls the contribution of entropy. —Parameters of the ResNet18 model; —Parameters of a temporal feature extraction network based on parallel-dense connections and the BGRU model; —BGRU model and fully connected layer parameters; —CTC objective function; —Modal alignment objective function.

4. The method according to claim 1, characterized in that, The steps of obtaining the continuous sign language recognition dataset and fine-tuning the dataset using a pre-trained BERT model to extract features of sign language words with action invariance include: Construct a BERT model and pre-train it to obtain sign language word sequences. Furthermore, the pre-trained BERT model was fine-tuned using a language model training strategy that shields the language model by employing sign language word sequences; The BERT model consists of twelve stacked Transformer modules; ； ； ； These are the input and output features of the Tranformer module, respectively. —Normalization layer; —Forward propagation layer; —Word embedding layer; —Position encoder; —Attention mechanism layer.

5. The method according to claim 1, characterized in that, The specific structure of each residual module in ResNet18 is as follows: ； ； —Batch normalization layer; —ReLU activation function; — Convolution kernel is Two-dimensional convolutional layers; ； video Spatial information characteristics — The output representation of the global average pooling layer.

6. The method according to claim 1, characterized in that, After the step of training the sensory feature extraction model to form a cross-modal feature alignment model, the method further includes: The cross-modal feature alignment model was tested. Specifically, CTC Beamsearch was used to predict the sign language visual features output by the model into sign sentences, and the WER metric was used to evaluate the predicted sentence, as shown in the following formula: ； —The number of insert, delete, and replace operations; —The number of words in the standard reference sentence.

7. A continuous sign language recognition device based on prior guidance of sign language words, characterized in that, include: Feature extraction module: used to acquire continuous sign language recognition dataset, fine-tune the continuous sign language recognition dataset using a pre-trained BERT model to extract sign language word features with action invariance, and construct a visual feature extraction model to extract sign language visual features. The continuous sign language recognition dataset includes continuous sign language recognition videos and corresponding sign language word sequences. Cross-modal feature alignment model module: used to construct cross-modal feature alignment objective function and CTC objective function to form objective function, add the objective function to pre-trained BERT model and visual feature extraction model to form cross-modal feature alignment model and train the cross-modal feature alignment model; The visual feature extraction model includes ResNet18, a parallel-dense connection-based temporal feature extraction network, a BGRU model, and a fully connected layer. The ResNet18 consists of 4 residual modules and 1 global average pooling layer; The modality feature alignment objective function includes: Construct the cost function: ; —It possesses the characteristics of sign language words with motion invariance; —Visual features of sign language; —Cost function; ； ； In the formula, ; ； ； — Discrete category probability distribution; — Discrete category probability distribution; —A hyperparameter that controls the contribution of entropy; = ； ； The parallel-dense connection-based temporal feature extraction network includes Densely connected blocks stacked in parallel; No. The first SRC block in The operations of the layer RDC are as follows: ； and —No. The first SRC block in Input and output characteristics of layered RDCs; — Convolution operation; —Refers to the weights and biases of the dilated convolutional layer; Refers to the feature dimension; Reuse In each SRC block Characteristics of layered RDCs: 。

Images