Sign language translation method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a sign language translation method and device, equipment, a storage medium and a product. The method comprises the steps that feature extraction is conducted on interaction data of multiple modes generated in the sign language interaction process, and interaction features of all the modes are obtained; performing time sequence modeling on the interaction features of the first modals to obtain a mixed feature sequence; wherein the first mode is a mode in which interaction data continuously changes along with time in multiple modes; performing cross-modal alignment on the interaction features of the second modal and the mixed feature sequence to obtain a multi-modal fusion feature vector; wherein the second modality is a modality, except the first modality, in the multiple modalities; and performing text sequence conversion on the multi-modal fusion feature vector to obtain a sign language translation result. According to the embodiment of the invention, comprehensive capture of dynamic and static features of the sign language is realized, the uniqueness of each mode is fully utilized, and the comprehensiveness and accuracy of sign language translation are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a sign language translation method and device, equipment, a storage medium and a product. BACKGROUND

[0002] Modern sign language translation technology mainly relies on deep neural networks to process multi-modal data of sign language, extracts features of different types of data and classifies them, aiming to realize real-time and accurate communication between hearing-impaired people and others. However, the existing technology has significant defects: first, it mainly relies on image data or its variants, which is difficult to fully capture the static and dynamic features of sign language; second, it lacks specialized processing procedures for different modal data, affecting the comprehensiveness and accuracy of sign language understanding. SUMMARY

[0003] The embodiments of the present application provide a sign language translation method, device, equipment, storage medium and product, which can solve the technical problem of breaking through the dependence of existing technology on single modal data and improving the comprehensiveness and accuracy of sign language understanding.

[0004] In a first aspect, the embodiments of the present application provide a sign language translation method, which comprises: respectively extracting features of multi-modal interaction data generated in a sign language interaction process to obtain interaction features of each modality; time series modeling of the interaction features of each first modality to obtain a mixed feature sequence; wherein the first modality is a modality in which interaction data changes continuously over time among the multi-modalities; cross-modal alignment of the interaction features of a second modality and the mixed feature sequence to obtain a multi-modal fusion feature vector; wherein the second modality is a modality other than the first modality among the multi-modalities; text sequence conversion of the multi-modal fusion feature vector to obtain a sign language translation result.

[0005] In an implementable embodiment, the step of respectively extracting features of multi-modal interaction data generated in a sign language interaction process to obtain interaction features of each modality comprises: obtaining multi-modal interaction data generated in a sign language interaction process; determining a feature extraction model of each interaction data according to the modality type of the interaction data; inputting the interaction data into the corresponding feature extraction model to obtain interaction features of each modality.

[0006] In an implementable embodiment, the types of the plurality of modalities generated in the sign language interaction process include a first modality and a second modality, the first modality includes a visual action modality and a physiological action modality; wherein the first modality is a modality in which interaction data continuously changes over time among the plurality of modalities; and the second modality is a modality other than the first modality among the plurality of modalities. The step of determining a feature extraction model of each interaction data according to the modality type of the interaction data includes: In the case where the modality type of the interaction data is the visual action modality, the feature extraction model of the interaction data is determined to be an efficient network encoder; wherein the efficient network encoder includes a plurality of inverted residual blocks, and the inverted residual block includes a high-efficiency channel attention module. In the case where the modality type of the interaction data is the physiological action modality, the feature extraction model of the interaction data is determined to be an autoencoder; wherein the autoencoder includes an encoder network and a decoder network. In the case where the modality type of the interaction data is the second modality, the feature extraction model of the interaction data is determined to be a bidirectional encoder representation model; wherein the bidirectional encoder representation model includes a plurality of stacked self-attention mechanism models.

[0007] In an implementable embodiment, the step of inputting the interaction data of each modality into the corresponding feature extraction model to obtain the interaction features of each modality includes: The interaction data of the visual action modality is input into the efficient network encoder, and the interaction data of the visual action modality is subjected to feature extraction and dimension change according to the inverted residual block in the efficient network encoder to obtain a first feature vector of the visual action modality. The first feature vector of the visual action modality is subjected to convolution operation to obtain a second feature vector of the visual action modality. The second feature vector of the visual action modality is subjected to flattening operation to obtain a third feature vector of the visual action modality. The third feature vector of the visual action modality is subjected to feature mapping according to the fully connected layer in the efficient network encoder to obtain the interaction features of the visual action modality.

[0008] In an implementable embodiment, the step of performing time series modeling on the interaction features of each first modality to obtain a mixed feature sequence includes: A dynamic complementarity matrix is constructed according to the interaction feature matrix of each first modality; wherein the dynamic complementarity matrix is used to represent the complementarity degree of the interaction features of different first modalities at each time step. The dynamic complementarity matrix is subjected to weighted fusion to obtain a fused feature. The fusion features are dynamically adjusted in time sequence to obtain adjusted fusion features. The adjusted fusion features are modeled by hierarchical spatiotemporal attention to obtain a mixed feature sequence.

[0009] In an implementable embodiment, the step of performing cross-modal alignment on the interaction features of the second modality and the mixed feature sequence to obtain a multi-modal fusion feature vector includes: The interaction features of the second modality and the mixed feature sequence are aligned by feature dimension reduction to obtain second-modality-aligned interaction features and aligned mixed feature sequence. The second-modality-aligned interaction features and the aligned mixed feature sequence are learned by contrast correlation to obtain second-modality-optimized interaction features and optimized mixed feature sequence. The second-modality-optimized interaction features and the optimized mixed feature sequence are input into a contrast learning model for feature fusion to obtain a multi-modal fusion feature vector.

[0010] In a second aspect, the embodiments of the present application provide a sign language translation device, which includes: An extraction module is configured to extract features from interaction data of multiple modalities generated in a sign language interaction process to obtain interaction features of each modality. A modeling module is configured to model time series of the interaction features of each first modality to obtain a mixed feature sequence, wherein the first modality is a modality in which interaction data of the multiple modalities changes continuously over time. An alignment module is configured to perform cross-modal alignment on the interaction features of a second modality and the mixed feature sequence to obtain a multi-modal fusion feature vector, wherein the second modality is a modality other than the first modality in the multiple modalities. A conversion module is configured to convert the multi-modal fusion feature vector into a text sequence to obtain a sign language translation result.

[0011] In a third aspect, the embodiments of the present application provide a sign language translation device, which includes a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the sign language translation method of the first aspect.

[0012] In a fourth aspect, the embodiments of the present application provide a computer storage medium, which stores computer program instructions readable by a computer, and the computer program instructions are executed by a processor to implement the sign language translation method of the first aspect.

[0013] In a fifth aspect, an embodiment of the present application provides a computer program product. Instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the sign language translation method according to the first aspect.

[0014] The sign language translation method, device, equipment, storage medium and product provided by the embodiments of the present application guarantee the independence, integrity and comprehensiveness of the original information by extracting the interaction features of each modality respectively; the dynamic evolution law is accurately captured by performing time sequence modeling on the modality features that change continuously over time, and a mixed feature sequence is obtained; then the cross-modality alignment mechanism is introduced to associate and adapt the time sequence features of the first modality with the non-time sequence features of the second modality in the shared semantic space, so as to realize efficient fusion and complementarity of different modality information and form a unified multi-modal fusion feature vector; finally, the translation result is obtained through text sequence conversion, which significantly improves the richness and discriminability of the sign language feature representation, realizes comprehensive capture of the static and dynamic features of the sign language, fully utilizes the uniqueness of each modality, and significantly improves the comprehensiveness and accuracy of sign language translation. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced. For those skilled in the art, other drawings can also be obtained without creative labor on the premise of these drawings.

[0016] Figure 1 is an embodiment one flowchart of a sign language translation method provided by the embodiments of the present application; Figure 2 is an embodiment two flowchart of a sign language translation method provided by the embodiments of the present application; Figure 3 is an autoencoder structure diagram of the sign language translation method provided by the embodiment two of the present application; Figure 4 is a BERT model structure diagram of the sign language translation method provided by the embodiment two of the present application; Figure 5 is a whole framework diagram of the sign language translation method provided by the embodiment two of the present application; Figure 6 is an embodiment three flowchart of a sign language translation method provided by the embodiments of the present application; Figure 7 is an EfficientNet encoder structure diagram of the sign language translation method provided by the embodiment three of the present application; Figure 8 is an inverted residual block structure diagram of the sign language translation method provided by the embodiment three of the present application; Figure 9is a structural schematic diagram of a sign language translation device provided by an embodiment of the present application. Figure 10 is a structural schematic diagram of a sign language translation device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purposes, technical solutions and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of the specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0018] It should be noted that, in this document, relational terms such as first and second and the like can only be used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed or other elements inherent to such process, method, article or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0019] The existing sign language translation technology has the following defects: 1. It mainly relies on image data or its variants, and it is difficult to fully capture the static and dynamic features of sign language; 2. It lacks specialized processing procedures for different modal data, affecting the comprehensiveness and accuracy of sign language understanding.

[0020] In order to solve the problems in the prior art, the embodiments of the present application provide a sign language translation method, device, equipment, storage medium and product. First, the sign language translation method provided by the embodiments of the present application will be introduced.

[0021] Figure 1 The flowchart of the sign language translation method provided by an embodiment of the present application is shown. As shown in Figure 1 The method can include steps 210-240: Step 210, respectively extracting features of the interaction data of multiple modalities generated in the sign language interaction process to obtain the interaction features of each modality.

[0022] The execution subject of the embodiment can be a sign language translation device, which is a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc. The sign language translation device can extract features of various modalities of interaction data generated in the sign language interaction process, perform time series modeling and cross-modal alignment based on the extracted interaction features of each modality, and obtain a multi-modal fusion feature vector; and convert the multi-modal fusion feature vector into a text sequence to obtain the corresponding sign language translation result.

[0023] The sign language interaction process refers to the process from when the sign language translation function is triggered to when all modalities of interaction data related to sign language are recorded. Alternatively, the types of various modalities generated in the sign language interaction process include but are not limited to a first modality and a second modality. The first modality refers to a modality whose interaction data changes continuously over time, and the first modality includes but is not limited to a visual action modality and a physiological action modality. The second modality is a modality other than the first modality among the various modalities generated in the sign language interaction process, which can be a non-sequential or semantic modality, and is used to provide context semantic information, such as text prompt data, scene classification data, etc. The scene classification data is used to represent the environmental context in which the sign language interaction occurs to supplement the environmental factors expressed by the sign language user, and its data form includes but is not limited to text, image, etc. For example, when communicating in a store, the "store" is a typical scene environment, and the corresponding typical feature can be asking the price.

[0024] The visual action modality is a sign language related modality type generated in the sign language interaction process and obtained through visual perception. The interaction data of the visual action modality can intuitively reflect the external action form of sign language expression, and the data presents continuous time sequence changes with the development of sign language action, which is a key modality for capturing dynamic features and static forms of sign language. The interaction data of the visual action modality includes but is not limited to continuous gesture image sequences and continuous body posture image sequences.

[0025] The physiological action modality is a physiological signal type modality generated in the sign language interaction process and directly related to the execution of sign language action. The interaction data of the physiological action modality can reflect the physiological activity state when the sign language action is executed, and the data presents continuous dynamic changes with the start, execution and end of the sign language action. The interaction data of the physiological action modality includes but is not limited to electromyographic signals, neural electrical signals, etc. In this embodiment, the interaction data of the physiological action modality is taken as an example to be described.

[0026] Optionally, in the embodiment, the following interaction data generated in the sign language interaction process is taken as an example for illustration: a continuous gesture image sequence, a continuous body posture image sequence, an electromyography signal, text prompt data, and scene classification data. The hand gestures and body gestures of the sign language user can be captured in real time by a high-precision infrared imaging device built in the sign language translation device or an external high-precision infrared imaging device, and the flexible changes of the fingers are accurately recorded to obtain the continuous gesture image sequence, and the information including but not limited to the spatial trajectory of the arm and the joint angle change is accurately recorded to obtain the continuous body posture image sequence. Meanwhile, the fine electromyography sensor is deployed in the key areas of the face of the sign language user, for example, near the temples, to capture the fine muscle activity electrical signals generated by the sign language user in the process of generating the hand gestures and body gestures, thereby obtaining the electromyography signal that adds an additional expression dimension to the sign language expression.

[0027] The text prompt data accurately corresponds to the semantics of the sign language action, for example, the concept of a large number of people is explained as rotating a circle with the fingers of both hands forming a figure, and the text prompt data provides an important reference for subsequent feature extraction and semantic understanding. The text prompt data and the scene classification data can be directly input by the sign language user through the sign language translation device, or other collection methods can be used, which are not limited in the embodiment.

[0028] After obtaining the interaction data of multiple modalities generated in the sign language interaction process, the interaction data of each modality is subjected to feature extraction to obtain interaction features that can reflect the core attributes of the corresponding modality. In the embodiment, different ways of feature extraction can be selected for the interaction data of different modalities. For example, the convolutional neural network is used to extract the interaction features of the visual action modality interaction data, and the Transformer encoder is used to extract the interaction features of the physiological action modality interaction data. The embodiment is not limited to the way of feature extraction.

[0029] In step 220, the interaction features of each first modality are subjected to time sequence modeling to obtain a mixed feature sequence; wherein the first modality is a modality in which the interaction data continuously changes with time among the multiple modalities.

[0030] After obtaining the interaction features of all modalities, the modality in which the interaction data continuously changes over time in the multiple modalities in the sign language interaction process is screened out as the first modality. Time series modeling is performed on all the interaction features of the first modality (for example, the interaction features corresponding to the continuous gesture image sequence in the visual action modality, the interaction features corresponding to the continuous body posture image sequence in the visual action modality, and the interaction features corresponding to the electromyographic signal in the physiological action modality), and the time sequence correlation between all the interaction features of the first modality is analyzed, so as to obtain a mixed feature sequence. In this embodiment, the mixed feature sequence is the fusion time sequence feature obtained after time series modeling of all the interaction features of the first modality, which integrates the dynamic information of different first modalities and retains the time sequence continuity.

[0031] Optionally, the long short-term memory network can be used to perform time series modeling on the interaction features of each first modality, and the long-time dependence is captured through the gating mechanism to splice the output mixed feature sequence. Other ways can also be used for time series modeling, which are not limited in this embodiment.

[0032] In a feasible implementation, step 220 can include steps A11-A13: Step A11, constructing a dynamic complementarity matrix according to the interaction feature matrix of each first modality; wherein the dynamic complementarity matrix is used to represent the complementarity degree of the interaction features of different first modalities at each time step.

[0033] Step A12, performing weighted fusion on the dynamic complementarity matrix to obtain a fusion feature.

[0034] Step A13, performing time sequence dynamic adjustment on the fusion feature to obtain an adjusted fusion feature.

[0035] Step A14, performing hierarchical spatiotemporal attention modeling on the adjusted fusion feature to obtain a mixed feature sequence.

[0036] Optionally, a dynamic complementary fusion network can be introduced to perform time series modeling on the interaction features of all the first modalities, and the specific process can be: constructing a dynamic complementarity matrix based on the interaction feature matrix of all the first modalities , which is used to represent the complementarity degree of different first modalities at each time step. For example, when there are interaction features corresponding to the continuous gesture image sequence in the visual action modality, interaction features corresponding to the continuous body posture image sequence in the visual action modality, and interaction features corresponding to the electromyographic signal in the physiological action modality, the dynamic complementarity matrix . Wherein, ∈ R^(3×3), is a Sigmoid function, is a trainable parameter matrix whose dimension depends on the dimension of the input features and the required output dimension, used to transform the features of different modalities into a common feature space, allowing the model to learn how to optimally combine these features to calculate complementarity. respectively are the interaction features corresponding to the sequence of continuous gesture images, the sequence of continuous body posture images, and the electromyography signal at time t.

[0037] The weighted fusion is performed based on the complementarity matrix, and the output result after the weighted fusion is the fusion feature . For example, when there are interaction features corresponding to the sequence of continuous gesture images in the visual action modality, interaction features corresponding to the sequence of continuous body posture images in the visual action modality, and interaction features corresponding to the electromyography signal in the physiological action modality, wherein represents the dynamic complementarity matrix the element in the first row and the first column, represents the element point multiplication of the interaction features corresponding to the sequence of continuous gesture images and the interaction features corresponding to the sequence of continuous body posture images, and the other parameters in the formula have similar meanings as described above.

[0038] The fusion feature is dynamically adjusted in time sequence, and optionally, a time sequence gating unit is introduced to dynamically adjust the fusion feature, thereby obtaining the adjusted fusion feature . The adjustment process is as follows: wherein is a trainable parameter matrix used to learn how to integrate the fusion features of the current time step and the previous time step, is a bias term, which, together with , introduces an offset in the calculation of the time sequence gating unit, helping the model to more flexibly learn the dynamic adjustment rule of the time sequence features.

[0039] Hierarchical spatio-temporal attention modeling is performed on the adjusted fusion feature, and spatial and temporal attention mechanisms are applied, thereby obtaining a mixed feature sequence , and the specific process is as follows: The full connection layer and the ReLU activation function are used to perform nonlinear transformation on , the LSTM (Long Short-Term Memory) layer is used to process sequence information, and then the feature is mapped to the output space, and finally the softmax function is used to convert the output into a probability distribution , specifically wherein is the trainable weight matrix of the output layer, which maps the output features of the LSTM layer to the final probability distribution space, whose dimension is determined by the LSTM output dimension and the number of classification categories; is the trainable weight matrix of the hidden layer, which performs linear transformation on , whose dimension is determined by the dimension of and the input dimension of the subsequent LSTM layer; is the bias term of the hidden layer, which cooperates with to introduce a learnable offset for the linear transformation before ReLU activation, improving the model's fitting flexibility for features; is the bias term of the output layer, which cooperates with to introduce a learnable offset for the linear transformation before softmax activation, helping the model to output probability distribution more accurately.

[0040] Optionally, a separate expert model is trained for each modality, which is transferred to the fusion model through knowledge distillation, and the distillation loss is calculated as follows: ; where KL is the Kullback-Leibler divergence, which measures the difference between two probability distributions; is the output probability distribution of the i-th expert model, in this embodiment, there are three expert models, namely gesture expert model, body posture expert model, and electromyographic signal expert model, each of which is individually pre-trained for a specific modality.

[0041] Learnable loss weights are introduced to balance the main task loss and the distillation loss, and the overall loss is calculated as follows: ; where is the main task loss, which evaluates the accuracy of the model in converting continuous sign language action sequences into target language text; and are learnable parameters. For example, when translating hello, the gesture expert may focus on the shape and movement of the hand; the body posture expert may notice the inclination of the upper body; the electromyographic signal expert may capture the facial muscle activity related to smiling; the fusion model learns how to integrate these information for accurate translation through the main task loss, while learning the unique insights of each expert model through the distillation loss.

[0042] In the embodiment, the dynamic complementary fusion network is introduced to model the time series of the interaction features of all first modalities, the dynamic complementary fusion network is carried through a Transformer framework, the hierarchical spatiotemporal attention is used to process the multi-modal input, the features are mapped to a probability space, the overall loss (including the main task loss, the knowledge distillation loss and the regularization loss) is used to provide a learning signal for the Transformer to guide the parameter optimization, the complex multi-modal data is effectively learned and processed, the knowledge is transferred from the pre-training model, the complex spatiotemporal dependency in the sign language translation is effectively captured and integrated, and the translation accuracy and robustness are significantly improved.

[0043] In step 230, the interaction features of the second modalities and the mixed feature sequence are cross-modally aligned to obtain a multi-modal fusion feature vector; wherein the second modalities are modalities other than the first modality among the plurality of modalities.

[0044] After obtaining the interaction features of all modalities, the modalities other than the first modality among the plurality of modalities in the sign language interaction process are selected as the second modalities. The interaction features of all second modalities and the mixed feature sequence are cross-modally aligned, i.e., the interaction features of all second modalities and the mixed feature sequence are projected into a shared semantic space, the feature heterogeneity between the second modalities and the first modality is eliminated through correlation adaptation, the semantic matching of information of different modalities is achieved, and thus a multi-modal fusion feature vector is obtained. In the embodiment, the multi-modal fusion feature vector refers to a unified feature identifier obtained after cross-modal alignment, which integrates information of each modality and has complementary advantages of time-series dynamic features and non-time-series static features.

[0045] Optionally, the semantic similarity between the second modality features and the mixed feature sequence can be calculated through an attention mechanism, and the multi-modal fusion feature vector can be obtained after weighted adaptation and projection into a unified space. Other ways can also be used for cross-modal alignment, which is not limited in the embodiment.

[0046] In a feasible embodiment, step 230 can include steps B11-B13: In step B11, the interaction features of the second modalities and the mixed feature sequence are aligned in feature dimension reduction to obtain the aligned interaction features of the second modalities and the aligned mixed feature sequence.

[0047] In step B12, the aligned interaction features of the second modalities and the aligned mixed feature sequence are subjected to comparative correlation learning to obtain the optimized interaction features of the second modalities and the optimized mixed feature sequence.

[0048] In step B13, the optimized interaction features of the second modalities and the optimized mixed feature sequence are input into a comparative learning model for feature fusion to obtain a multi-modal fusion feature vector.

[0049] Optionally, the interaction features and the mixed feature sequences of all the second modalities can be cross-modally aligned in a contrastive learning manner. Specifically, the interaction features (e.g., the interaction features corresponding to the text prompt data, the interaction features corresponding to the scene classification data) and the mixed feature sequences of all the second modalities can be projected into a shared low-dimensional feature space by principal component analysis, so as to realize effective comparison and fusion of the features of different modalities, achieve the purpose of feature dimension reduction and alignment, and obtain the aligned interaction features of the second modalities and the aligned mixed feature sequences.

[0050] The aligned interaction features of the second modalities are taken as positive samples, and the aligned mixed feature sequences are taken as negative samples to obtain a positive-negative sample pair. The semantic matching degree between the positive-negative sample pair is strengthened and the modal heterogeneity difference is reduced by contrastive correlation learning, so as to obtain the optimized interaction features of the second modalities and the optimized mixed feature sequences.

[0051] The contrastive learning model adopts a Siamese network as the basic architecture of contrastive learning. The contrastive learning model mainly consists of two key components: a feature extraction layer and a fusion layer. The optimized interaction features of the second modalities and the optimized mixed feature sequences are input into the feature extraction layer in the contrastive learning model. The feature extraction layer is responsible for extracting key features from the optimized interaction features of the second modalities and the optimized mixed feature sequences. The fusion layer integrates the key features of different modalities output by the feature extraction layer into a unified feature representation, thereby outputting a multi-modal fusion feature vector.

[0052] In this embodiment, the alignment between the features of different modalities is achieved by contrastive learning, and the semantic correlation between the information of different modalities is established.

[0053] In step 240, the multi-modal fusion feature vector is subjected to text sequence conversion to obtain a sign language translation result.

[0054] After obtaining the multi-modal fusion feature vector, the multi-modal fusion feature vector is subjected to text sequence conversion, and the abstract multi-modal fusion feature vector is mapped to a natural language text, thereby obtaining an understandable sign language translation result.

[0055] Optionally, a decoder text decoder based on a Transformer (self-attention mechanism model) architecture can be used to convert the multi-modal fusion feature vector into a text sequence in a target language, complete the conversion process from sign language to text, and thereby obtain a sign language translation result. The decoder text decoder based on the Transformer architecture includes multiple decoder layers, and utilizes multi-head attention mechanism, feedforward neural network, residual connection and normalization to realize efficient and stable translation.

[0056] The embodiment extracts interaction features of each modality respectively, guarantees the independence, integrity and comprehensiveness of the original information, models the modality features changing continuously with time, accurately captures the dynamic evolution law, and obtains a mixed feature sequence. Then, a cross-modality alignment mechanism is introduced to associate and adapt the time sequence features of the first modality with the non-time sequence features of the second modality in a shared semantic space, efficiently fuses and complements different modality information, and forms a unified multi-modal fusion feature vector. Finally, the translation result is obtained through text sequence conversion, which significantly improves the richness and discriminability of sign language feature representation, realizes comprehensive capture of sign language dynamic and static features, fully utilizes the uniqueness of each modality, and significantly improves the comprehensiveness and accuracy of sign language translation.

[0057] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 2 , in the sign language translation method, step 210 can include steps 310-330: Step 310, acquiring interaction data of multiple modalities generated in the sign language interaction process.

[0058] Step 320, determining the feature extraction model of each interaction data according to the modality type of each interaction data.

[0059] Step 330, inputting each interaction data into the corresponding feature extraction model to obtain the interaction features of each modality.

[0060] In the embodiment, the modality type includes two main types of first modality and second modality, and the first modality includes visual action modality and physiological action modality. The classification can also be adjusted according to the actual sign language interaction scene and the demand of new modalities, which is not limited in the embodiment.

[0061] The interaction data of different modality types correspond to different feature extraction models, and the model needs to be adapted to the structure and feature law of the interaction data.

[0062] Optionally, the interaction data of multiple modalities is collected through the built-in device or external special collection device of the sign language translation device, including but not limited to continuous gesture image sequence of visual action modality, continuous body posture image sequence of visual action modality, electromyographic signal of physiological action modality, text prompt data of second modality and scene classification data of second modality.

[0063] Based on the modality type of each interaction data, the feature extraction model of each interaction data is matched, each interaction data is processed by the feature extraction model of each interaction feature, and the interaction features of each modality are output.

[0064] In an implementable embodiment, the types of the plurality of modalities generated in the sign language interaction process include a first modality and a second modality, the first modality includes a visual action modality and a physiological action modality; wherein the first modality is a modality in which the interaction data continuously changes over time among the plurality of modalities; the second modality is a modality other than the first modality among the plurality of modalities; step 320 can include steps C11-C13: Step C11, in the case where the modality type of the interaction data is a visual action modality, determining that the feature extraction model of the interaction data is an efficient network encoder; wherein the efficient network encoder includes a plurality of inverted residual blocks, and the inverted residual block includes an efficient channel attention module.

[0065] In the case where the modality type of the interaction data is a visual action modality, the feature extraction model of the interaction data is determined to be an efficient network encoder. For example, the feature extraction model corresponding to the continuous gesture image sequence and the continuous body posture image sequence is a pre-trained efficient network encoder (EfficientNet encoder). Wherein the core construction of the efficient network encoder is a modified inverted residual block (MBConv), and each modified inverted residual block includes an ECA (Efficient Channel Attention, efficient channel attention mechanism) module.

[0066] Step C12, in the case where the modality type of the interaction data is a physiological action modality, determining that the feature extraction model of the interaction data is an autoencoder; wherein the autoencoder includes an encoder network and a decoder network.

[0067] In the case where the modality type of the interaction data is a physiological action modality, the feature extraction model of the interaction data is determined to be an autoencoder including an encoder network and a decoder network. For example, the feature extraction model corresponding to the electromyographic signal is an autoencoder.

[0068] Wherein, taking the electromyographic signal as an input of the autoencoder as an example, the structure of the autoencoder is described, and a structural diagram of the autoencoder is as follows: Figure 3As shown, the encoder network has multiple convolutional layers for receiving and processing the original surface electromyography signal, and the encoder network comprises: a first convolutional unit including two consecutive convolutional layers, each convolutional layer having a convolution kernel size of 3x3, and each convolutional layer being followed by a ReLU activation function; a second convolutional unit including one convolutional layer having a convolution kernel size of 5x5, the convolutional layer being followed by a ReLU activation function; and a third convolutional unit including one convolutional layer having a convolution kernel size of 5x5, the convolutional layer being followed by a Sigmoid activation function. The decoder network is used to process the deep features of the electromyography signal generated by the encoder network, and the decoder network comprises: a first deconvolutional unit including two consecutive deconvolutional layers, each deconvolutional layer having a deconvolution kernel size of 5x5, and each deconvolutional layer being followed by a ReLU activation function; a second deconvolutional unit including one deconvolutional layer having a deconvolution kernel size of 3x3, the deconvolutional layer being followed by a ReLU activation function; and a third deconvolutional unit including one deconvolutional layer having a deconvolution kernel size of 3x3, the deconvolutional layer being followed by a Sigmoid activation function.

[0069] Through the above structure, the encoder network can extract deep features from the original electromyography signal; and the decoder network realizes the reconstruction of the electromyography signal. When the mean square error between the electromyography signal reconstructed by the decoder network and the original electromyography signal is less than a preset threshold, the deep features of the electromyography signal output by the encoder network are taken as the interaction features of the electromyography signal; otherwise, the structure of the encoder network is adjusted until the mean square error between the electromyography signal reconstructed by the decoder network and the original electromyography signal is less than the preset threshold.

[0070] In a case where the modality type of the interaction data is the second modality, the feature extraction model of the interaction data is determined as a bidirectional encoder representation model. The bidirectional encoder representation model includes a plurality of stacked self-attention mechanism models.

[0071] In a case where the modality type of the interaction data is the second modality, the feature extraction model of the interaction data is determined as a BERT (Bidirectional Encoder Representations from Transformers) model. The BERT model is obtained through training of a large-scale text data set, and the encoder structure of the BERT model is stacked by a plurality of same Transformer (self-attention mechanism model) encoder layers. Each Transformer encoder layer includes a multi-head attention mechanism and a feedforward neural network. For example, the feature extraction model corresponding to the scene classification data and the text prompt data is the BERT model.

[0072] Using text prompt data as input to the BERT model as an example, the structure of the BERT model is explained as follows: Figure 4 As shown in the diagram, word embedding converts each word in the input text into a high-dimensional vector representation; paragraph embedding uses embedding vectors to distinguish different text paragraphs or sequences; positional embedding provides the positional information of words in the text. Since the Transformer architecture itself does not have the ability to capture sequence order, it generates positional embedding vectors to provide the system with the positional information of words in the text; embedding summation performs element-level summation of the above three embedding vectors to generate a comprehensive representation vector that integrates multiple information, forming the final input representation; aggregation output performs global information aggregation on the word features output from the final encoder layer to obtain the overall semantic representation of the text, used to capture global text information; sequence output retains the feature sequence of each word output from the final encoder layer to capture the local contextual information of each word in the text, adapting to the requirements of sequence-level tasks; finally, the aggregation output and sequence output are input into the classifier to complete the feature extraction of the text prompt data and obtain the interaction features of the corresponding modality.

[0073] In this embodiment, the EfficientNet encoder is used to extract features from the interaction data of the visual action modality, the autoencoder is used to extract features from the interaction data of the physiological action modality, and the BERT model is used to extract features from the interaction data of the second modality. This achieves comprehensive information collection while ensuring the accuracy of feature extraction. In some embodiments, continuous gesture image sequences, continuous body posture image sequences, electromyographic signals, text prompt data, and scene classification data are used as examples of multimodal interaction data generated during sign language interaction to illustrate the overall process of sign language translation. The overall framework is as follows: Figure 5 As shown, the EfficientNet encoder was used to extract interaction features from continuous gesture image sequences and continuous body posture image sequences, respectively; an autoencoder was used to extract interaction features from electromyography (EMG) signals; and a BERT model was used to extract interaction features from text prompt data and scene classification data, respectively. Then, a Transformer was used to perform time-series modeling of the interaction features of continuous gesture image sequences, continuous body posture image sequences, and EMG signals. Through contrastive learning, modal alignment and feature fusion of interaction features from mixed feature sequences, text prompt data, and scene classification data were achieved. Finally, a Transformer decoder was used to generate target language text sequences, realizing sign language recognition and translation, improving the naturalness and comprehensiveness of sign language translation; enhancing real-time translation capabilities; significantly improving the accuracy and reliability of sign language recognition; and greatly improving the quality of real-time communication between hearing-impaired individuals and others.

[0074] The embodiment obtains various modal interaction data generated in the sign language interaction process; determines a feature extraction model of each interaction data according to a modal type of the interaction data; and inputs the interaction data into the corresponding feature extraction model to obtain interaction features of each modal. By matching the corresponding feature extraction model for the interaction data of each modal type, the accuracy of feature extraction is effectively ensured.

[0075] Based on any one or more embodiments of the present application, in a third embodiment of the present application, the same or similar content as the above embodiments can be referred to the above introduction, and will not be described in detail. On this basis, please refer to Figure 6 , the step 330 can include steps 410-440: Step 410, input the interaction data of the visual action modal into the EfficientNet encoder, and perform feature extraction and dimension change on the interaction data of the visual action modal according to the inverted residual block in the EfficientNet encoder to obtain the first feature vector of the visual action modal.

[0076] Step 420, performing convolution operation on the first feature vector of the visual action modal to obtain the second feature vector of the visual action modal.

[0077] Step 430, performing flattening operation on the second feature vector of the visual action modal to obtain the third feature vector of the visual action modal.

[0078] Step 440, performing feature mapping on the third feature vector of the visual action modal according to the full connection layer in the EfficientNet encoder to obtain the interaction features of the visual action modal.

[0079] Optionally, the EfficientNet encoder is obtained by training a large-scale image data set, and the structure thereof is as shown in Figure 7 , including 8 modified inverted residual blocks MBConv, a convolution layer (1×1Conv) with a convolution kernel size of 1×1, a flattening (Flatten) operation and a full connection layer FC.

[0080] The interaction data of the visual action modal (such as a continuous gesture image sequence or a continuous body posture image sequence) is input into the EfficientNet encoder, and the modified inverted residual block in the EfficientNet encoder performs feature extraction and dimension change on the interaction data of the visual action modal, thereby obtaining the first feature vector of the visual action modal. The first feature vector is the feature vector output by the last modified inverted residual block in the EfficientNet encoder.

[0081] Optionally, the structure of the modified inverted residual block is as shown inFigure 8 As shown, the process of processing input data using a single improved inverted residual block includes the following steps: Step 1: Feature dimensionality enhancement using a 1×1 convolutional layer; applying Batch Normalization (BN) and the Swish activation function. Step 2: Feature extraction using a k×k depth convolution, where k is 3 or 5. Step 3: Using the ECA module. Step 4: Feature dimensionality reduction using an additional 1×1 convolutional layer. Step 5: Performing dropout to improve overfitting resistance and generalization performance.

[0082] In this implementation, an ECA module is used to replace the traditional SE (Squeeze-and-Excitation) mechanism. Its function is to dynamically adjust the weights of each channel in the feature map to improve the model's ability to perceive channel importance and enhance overall feature representation capabilities. Specifically, the ECA module first processes the data using global average pooling to obtain global channel information. The calculation formula is as follows: ,in, This indicates the first input dimension of size H×W. Let there be three image feature maps, where y represents the global feature, a is the position index in the height direction (vertical direction), and b is the position index in the width direction (horizontal direction). The channel dimension C and the number of cross channels z of the feature map satisfy the following formula: .in, Indicates separation The most recent odd number, Indicates the channel dimension. and In this embodiment, the constant is represented. , The above constants can also be adjusted according to requirements. Then, using a convolution kernel size of... One-dimensional convolution is used to calculate channel weights and obtain the interdependencies between channels. The formula for calculating channel weights is: .in, Indicates channel weight, This represents the Sigmoid function. This indicates that the kernel size is [value]. One-dimensional convolution operation, The result after global average pooling combines the original input image features with channel weights. Performing dot product operations to obtain features with channel attention can more effectively focus on image feature information and enhance image feature extraction capabilities.

[0083] The EfficientNet encoder can effectively improve the feature extraction and expression ability, reduce the computational complexity, and enhance the model generalization performance.

[0084] After obtaining the first feature vector, the first feature vector is subjected to a convolution operation by the convolution layer of the EfficientNet encoder to realize fusion and compression of feature channels, thereby obtaining a second feature vector; the second feature vector is subjected to a flattening operation to convert a multi-dimensional data structure (such as a two-dimensional matrix or a higher-dimensional tensor) into a one-dimensional data structure, thereby obtaining a third feature vector; and the third feature vector is input into the fully connected layer FC of the EfficientNet encoder to realize feature space mapping through linear transformation and a nonlinear activation function, thereby finally generating the interaction feature of the visual action modality.

[0085] The embodiment inputs the interaction data of the visual action modality into the EfficientNet encoder, extracts and changes the dimensions of the interaction data of the visual action modality according to the inverted residual block in the EfficientNet encoder, obtains a first feature vector of the visual action modality, performs a convolution operation on the first feature vector of the visual action modality to obtain a second feature vector of the visual action modality, performs a flattening operation on the second feature vector of the visual action modality to obtain a third feature vector of the visual action modality, and performs feature mapping on the third feature vector of the visual action modality according to the fully connected layer in the EfficientNet encoder to obtain the interaction feature of the visual action modality. The inverted residual block realizes efficient feature extraction, the channel optimization of the convolution, and the feature mapping of the fully connected layer, thereby ensuring the discriminability and robustness of the visual action modality feature, retaining spatial feature information and realizing deep abstraction of the feature, and providing a high-quality visual feature input for subsequent multi-modal fusion.

[0086] It should be noted that the acquisition, storage, use, and processing of data in the embodiments of the present application comply with relevant provisions of national laws and regulations. In addition, it should be noted that in the embodiments of the present application, some industry existing schemes such as software, components, and models may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but does not mean that the applicant has or will necessarily use the scheme.

[0087] As shown in FIG. 2, Figure 9 The hand sign translation device 200 can include an extraction module 201, a modeling module 202, an alignment module 203, and a conversion module 204. The extraction module 201 is configured to extract features of interaction data of multiple modalities generated in a hand sign interaction process to obtain interaction features of the modalities.

[0088] The modeling module 202 is configured to perform time sequence modeling on the interaction features of each first modality to obtain a mixed feature sequence; the first modality is a modality in which interaction data continuously changes over time in the plurality of modalities.

[0089] The alignment module 203 is configured to perform cross-modal alignment on the interaction features of the second modality and the mixed feature sequence to obtain a multi-modal fusion feature vector; the second modality is a modality other than the first modality in the plurality of modalities.

[0090] The conversion module 204 is configured to perform text sequence conversion on the multi-modal fusion feature vector to obtain a sign language translation result.

[0091] Optionally, the extraction module 201 is further configured to: obtain interaction data in a plurality of modalities generated in a sign language interaction process; determine a feature extraction model of each interaction data according to a modality type of the interaction data; input the interaction data into the corresponding feature extraction model respectively to obtain interaction features of each modality.

[0092] Optionally, the extraction module 201 is further configured to: in a case where the modality type of the interaction data is a visual action modality, determine that the feature extraction model of the interaction data is an efficient network encoder; wherein the efficient network encoder includes a plurality of inverted residual blocks, and the inverted residual block includes a high-efficiency channel attention module; in a case where the modality type of the interaction data is a physiological action modality, determine that the feature extraction model of the interaction data is an auto-encoder; wherein the auto-encoder includes an encoder network and a decoder network; in a case where the modality type of the interaction data is a second modality, determine that the feature extraction model of the interaction data is a bidirectional encoder representation model; wherein the bidirectional encoder representation model includes a plurality of stacked self-attention mechanism models.

[0093] Optionally, the extraction module 201 is further configured to: input the interaction data in the visual action modality into the efficient network encoder, and perform feature extraction and dimension change on the interaction data in the visual action modality according to the inverted residual blocks in the efficient network encoder to obtain a first feature vector of the visual action modality; perform convolution operation on the first feature vector of the visual action modality to obtain a second feature vector of the visual action modality; perform flattening operation on the second feature vector of the visual action modality to obtain a third feature vector of the visual action modality; The third feature vector of the visual action modality is mapped in feature according to a full connection layer in the efficiency network encoder, to obtain interaction features of the visual action modality.

[0094] Optionally, the modeling module 202 is further configured to: The dynamic complementarity matrix is constructed according to the interaction feature matrix of each first modality, and the dynamic complementarity matrix is used to represent the complementary degree of the interaction features of different first modalities at each time step. The dynamic complementarity matrix is weighted and fused to obtain a fused feature. The fused feature is adjusted in time sequence to obtain an adjusted fused feature. The adjusted fused feature is modeled in hierarchical spatiotemporal attention to obtain a mixed feature sequence.

[0095] Optionally, the alignment module 203 is further configured to: The interaction features of the second modality and the mixed feature sequence are aligned in feature dimension reduction to obtain the second modality aligned interaction features and the aligned mixed feature sequence. The second modality aligned interaction features and the aligned mixed feature sequence are learned in comparison and association to obtain the second modality optimized interaction features and the optimized mixed feature sequence. The second modality optimized interaction features and the optimized mixed feature sequence are input into a contrast learning model for feature fusion to obtain a multi-modal fusion feature vector.

[0096] Figure 10 A hardware structure schematic diagram of a sign language translation device provided by an embodiment of the present application is shown.

[0097] The sign language translation device can include a processor 301 and a memory 302 storing computer program instructions.

[0098] Specifically, the processor 301 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits that implement one or more embodiments of the present application.

[0099] The memory 302 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 302 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a tape drive, a USB drive, or a combination of two or more of these. In one example, the memory 302 can include removable or non-removable (or fixed) media, where the memory 302 is non-volatile solid state memory. The memory 302 can be internal or external to the integrated gateway disaster recovery appliance.

[0100] In one example, the memory 302 can be a read-only memory (ROM). In one example, the ROM can be a mask programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0101] The memory 302 can include read-only memory (ROM), random access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to perform operations described with reference to methods according to aspects of the present disclosure.

[0102] The processor 301 implements the hand sign translation method in the illustrated embodiment by reading and executing computer program instructions stored in the memory 302. Figure 1 The hand sign translation method in the illustrated embodiment.

[0103] In one example, the hand sign translation device can further include a communication interface 303 and a bus 304. Wherein, as shown, the processor 301, the memory 302, the communication interface 303 are connected through the bus 304 and complete the communication between each other. Figure 10

[0104] The communication interface 303 is mainly used to realize the communication between each module, device, unit and / or equipment in the embodiment of the application.

[0105] ​Bus 304 includes hardware, software, or both, to couple components of the sign language translation device to each other in communication. While Figure 1 provides an example of a bus, it is to be understood that other bus architectures can be used in conjunction with the various embodiments discussed herein. For example, an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these can be used in the various embodiments. Where appropriate, bus 304 can include one or more buses. Although the present embodiments describe and show a particular bus, the present embodiments contemplate any suitable bus or interconnect.

[0106] The sign language translation device can be based on a sign language translation method, thereby achieving the combination of Figures 1-8 The described sign language translation method.

[0107] In addition, in combination with the sign language translation method in the above embodiments, the present embodiments can provide a computer storage medium to implement. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any one of the sign language translation methods in the above embodiments.

[0108] The present embodiments also provide a computer program product, comprising a computer program, the computer program being executed by a processor to implement any one of the sign language translation methods in the above embodiments.

[0109] It needs to be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0110] The functions indicated in the structural block diagrams described above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium, or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine-readable medium" can include any medium that can store or transfer information. Examples of the machine-readable medium include an electronic circuit, a semiconductor memory device, a read-only memory (ROM), a flash memory, an erasable read-only memory (EROM), a floppy diskette, a compact disk read-only memory (CD-ROM), an optical disk, a hard disk, a fiber optic medium, a radio frequency (RF) link, and the like. The code segments can be downloaded via computer networks such as the Internet, an intranet, and the like.

[0111] It is also noted that the examples mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the steps mentioned in the examples, that is, the steps can be performed in the order mentioned in the examples, or in an order different from the examples, or several steps can be performed simultaneously.

[0112] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing device to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing device to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0113] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A sign language translation method, characterized by, The method comprises: respectively extracting features of multiple modal interaction data generated in the sign language interaction process to obtain interaction features of each modality; time series modeling is performed on the interaction features of each first modality to obtain a mixed feature sequence; wherein the first modality is a modality in which interaction data continuously changes over time among the multiple modalities; cross-modal alignment is performed on the interaction features of the second modality and the mixed feature sequence to obtain a multi-modal fusion feature vector; wherein the second modality is a modality other than the first modality among the multiple modalities; text sequence conversion is performed on the multi-modal fusion feature vector to obtain a sign language translation result.

2. The method of claim 1, wherein, The step of respectively extracting features of multiple modal interaction data generated in the sign language interaction process to obtain interaction features of each modality comprises: obtaining multiple modal interaction data generated in the sign language interaction process; determining the feature extraction model of each interaction data according to the modality type of each interaction data; inputting each interaction data into the corresponding feature extraction model to obtain the interaction features of each modality.

3. The method of claim 2, wherein, The types of multiple modalities generated in the sign language interaction process include a first modality and a second modality, the first modality includes a visual action modality and a physiological action modality; wherein the first modality is a modality in which interaction data continuously changes over time among the multiple modalities; the second modality is a modality other than the first modality among the multiple modalities; The step of determining the feature extraction model of each interaction data according to the modality type of each interaction data comprises: in the case that the modality type of the interaction data is a visual action modality, determining that the feature extraction model of the interaction data is an efficiency network encoder; wherein the efficiency network encoder comprises a plurality of inverted residual blocks, and the inverted residual block comprises a high-efficiency channel attention module; in the case that the modality type of the interaction data is a physiological action modality, determining that the feature extraction model of the interaction data is an autoencoder; wherein the autoencoder comprises an encoder network and a decoder network; in the case that the modality type of the interaction data is a second modality, determining that the feature extraction model of the interaction data is a bidirectional encoder representation model; wherein the bidirectional encoder representation model comprises a plurality of stacked self-attention mechanism models.

4. The method of claim 2, wherein, The step of inputting each interaction data into the corresponding feature extraction model to obtain the interaction features of each modality comprises: inputting the interaction data of the visual action modality into the efficiency network encoder, and performing feature extraction and dimension change on the interaction data of the visual action modality according to the inverted residual blocks in the efficiency network encoder to obtain a first feature vector of the visual action modality; performing convolution operation on the first feature vector of the visual action modality to obtain a second feature vector of the visual action modality; performing flattening operation on the second feature vector of the visual action modality to obtain a third feature vector of the visual action modality; performing feature mapping on the third feature vector of the visual action modality according to the full connection layer in the efficiency network encoder to obtain the interaction features of the visual action modality.

5. The method of claim 1, wherein, The step of performing time series modeling on the interaction features of each first modality to obtain a mixed feature sequence comprises: constructing a dynamic complementarity matrix according to the interaction feature matrix of each first modality; wherein the dynamic complementarity matrix is used to represent the complementary degree of the interaction features of different first modalities at each time step; performing weighted fusion on the dynamic complementarity matrix to obtain a fused feature; performing time series dynamic adjustment on the fused feature to obtain an adjusted fused feature; performing hierarchical spatio-temporal attention modeling on the adjusted fused feature to obtain a mixed feature sequence.

6. The method of claim 1, wherein, The step of performing cross-modal alignment on the interaction features of the second modality and the mixed feature sequence to obtain a multi-modal fusion feature vector comprises: performing feature dimension reduction alignment on the interaction features of the second modality and the mixed feature sequence to obtain the second modality aligned interaction features and the aligned mixed feature sequence; performing comparative correlation learning on the second modality aligned interaction features and the aligned mixed feature sequence to obtain second modality optimized interaction features and optimized mixed feature sequence; inputting the second modality optimized interaction features and the optimized mixed feature sequence into a comparative learning model for feature fusion to obtain a multi-modal fusion feature vector.

7. A sign language translation device, characterized by, The device comprises: an extraction module for extracting features from the interaction data of multiple modalities generated in the sign language interaction process to obtain interaction features of each modality; a modeling module for performing time series modeling on the interaction features of each first modality to obtain a mixed feature sequence; wherein the first modality is a modality in which the interaction data of the multiple modalities changes continuously over time; an alignment module for performing cross-modal alignment on the interaction features of the second modality and the mixed feature sequence to obtain a multi-modal fusion feature vector; wherein the second modality is a modality in the multiple modalities other than the first modality; a conversion module for converting the multi-modal fusion feature vector into a text sequence to obtain a sign language translation result.

8. A sign language interpreting device, characterized by, The sign language translation device comprises a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the sign language translation method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, which are executed by the processor to implement the sign language translation method of any one of claims 1-6.

10. A computer program product, characterised in that, The instructions in the computer program product are executed by the processor of the electronic device to cause the electronic device to perform the sign language translation method of any one of claims 1-6.

Images