A sign language speech mutual interpretation method and system based on deep learning

Abstract

This invention discloses a deep learning-based method and system for sign language and speech translation, specifically relating to the fields of artificial intelligence, multimodal information processing, and human-computer interaction. By collecting sign language video, speech, and text data, key hand points are extracted and temporal features are constructed. A temporal coding network is used for modeling, and the data is mapped to a unified semantic representation space through a cross-modal attention mechanism to generate a structured semantic intermediate representation. Based on this, the conversion from sign language to text and speech, as well as the generation of text or speech-to-sign language action sequences, are achieved, thus enabling bidirectional translation between multiple modalities. By introducing a unified semantic representation space and structured semantic intermediate representation, this invention improves the semantic alignment between sign language and speech / text, reduces information loss and ambiguity, enhances the ability to express continuous sign language actions, and improves the system's stability and applicability while ensuring recognition and generation accuracy.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, multimodal information processing and human-computer interaction, and in particular to a method and system for sign language and speech translation based on deep learning. Background Technology

[0002] Sign language, as an important means of communication for the hearing impaired, relies on multi-dimensional information such as hand gestures, spatial location, movement trajectories, and facial expressions. Compared with natural language, sign language has significant spatiotemporal characteristics, and its word order structure and semantic organization differ.

[0003] In existing technologies, sign language translation systems mainly fall into the following categories: rule-based or template-matching methods, which rely on predefined rules and are difficult to handle complex semantics; sign language recognition methods based on a single visual model, which only achieve one-way conversion from sign language to text and lack reverse generation capabilities; sign language recognition methods based on deep learning, which can extract features but usually lack a cross-modal semantic unification mechanism; existing systems mostly adopt independent module designs, and the lack of a unified semantic intermediate representation between modules leads to error accumulation; in the process of bidirectional translation, the semantic space is inconsistent, which easily leads to semantic drift problems.

[0004] In addition, existing methods generally suffer from the following technical defects: insufficient temporal modeling capabilities, making it difficult to capture long-term dependencies; lack of a unified representation structure between sign language and text; semantic inconsistency during bidirectional translation; and difficulty in deploying the system on edge devices.

[0005] Therefore, it is necessary to propose a new technical solution to address the above problems. Summary of the Invention

[0006] The main objective of this invention is to provide a deep learning-based sign language and speech translation method and system, which realizes bidirectional conversion between sign language and speech / text by constructing a unified semantic representation space.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A deep learning-based sign language-to-speech translation system includes: The multimodal acquisition module is used to acquire sign language video data, voice data, and text data; The visual feature extraction module is used to detect and track the hand area in the sign language video data, extract the two-dimensional or three-dimensional key point coordinate sequence of the hands in continuous video frames, and perform noise reduction, smoothing and time sequence alignment processing on the key point coordinate sequence to generate a standardized sign language key point action sequence. The sign language temporal coding module is used to input the standardized sign language key point action sequence into a temporal coding network containing a multi-head attention mechanism to extract the local dynamic features and global dependency features of the sign language actions in the time dimension and output the temporal coding features. A cross-modal semantic mapping module is used to map the temporal encoded features to a unified semantic representation space through a cross-modal attention mechanism. The unified semantic representation space is a semantic representation space based on vector embedding, and generates a structured sign language semantic intermediate representation. The structured sign language semantic intermediate representation includes at least one of the following information: sign language word sequence, action boundary marker, semantic role relationship, and word order dependency relationship. The natural language generation module is used to input the structured sign language semantic intermediate representation into a compressed and optimized neural network language model, reconstruct the word order, perform semantic completion and syntactic optimization, and generate the target text sequence. The speech processing module is used to realize bidirectional conversion between text data and speech data, including: converting the target text sequence into speech output, and converting the input speech into text to be parsed; The sign language action generation module includes: a text semantic encoding unit, used to encode the text to be parsed into a semantic feature vector and map it to the unified semantic representation space, wherein the text to be parsed includes text obtained by speech conversion or collected text data; An action sequence generation unit is used to generate a sign language key point action sequence or a sign language skeleton driven sequence through a sequence generation network based on the semantic representation in the unified semantic representation space. The interactive output module is used to output the target text sequence, speech output, and sign language visualization results driven by the sign language key point action sequence, so as to realize bidirectional translation between sign language and speech / text.

[0008] Preferably, the visual feature extraction module further includes a temporal normalization processing unit, which is used to align the temporal scales of sign language key point action sequences of different lengths and unify the sequence length through interpolation or resampling to improve the stability and robustness of subsequent temporal modeling.

[0009] Preferably, the temporal coding network is a combination of at least one of the following structures: bidirectional long short-term memory network, gated recurrent unit network, temporal convolutional network, and Transformer coding network, and enhances the ability to model long-term dependencies through a multi-layer stacked structure.

[0010] Preferably, the cross-modal semantic mapping module adopts a multi-head cross-modal attention mechanism, which realizes semantic alignment between sign language and text through a unified semantic representation space, and enhances the representation of key action segments through attention weights to improve the accuracy of semantic mapping.

[0011] Preferably, the neural network language model is a lightweight model that has undergone quantization, pruning, or knowledge distillation, and supports low-latency inference execution on resource-constrained devices.

[0012] Preferably, the action sequence generation unit is based on an autoregressive sequence generation model or a diffusion generation model, maps semantic feature vectors to continuous sign language key point action sequences, and optimizes the continuity of generated actions through smoothness constraints or motion consistency constraints.

[0013] A deep learning-based sign language speech translation method includes the following steps: S1: Collect sign language video data, voice data, and text data; S2: Perform hand detection, tracking and key point extraction on the sign language video data to obtain temporal data of hand key points, and perform noise reduction, smoothing and temporal alignment processing to generate a sequence of sign language key point actions; S3: Input the sign language key point action sequence into a temporal coding network containing a multi-head attention mechanism for encoding to obtain temporal coding features; S4: The temporal coding features are mapped to a unified semantic representation space through a cross-modal attention mechanism to generate a structured sign language semantic intermediate representation; S5: Input the structured sign language semantic intermediate representation into the neural network language model to generate the target text sequence, and convert the target text sequence into speech output; S6: Convert the input speech into text to be parsed, or acquire text data, and encode it into the unified semantic representation space through a text semantic coding model; S7: Generate a sequence of sign language key points based on the semantic representation in the unified semantic representation space, and output the corresponding sign language visualization results, thereby realizing bidirectional translation between sign language and speech / text.

[0014] Preferably, based on user geographic information, historical interaction data, and user feedback information, the temporal coding network, cross-modal semantic mapping model, and action generation model are adaptively updated to achieve dynamic adaptation to different sign language expressions; and The preprocessing of sign language video data, key point extraction, and some reasoning processes are deployed on the edge device to reduce the system's dependence on the network environment and to achieve basic translation functions under offline conditions.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a unified semantic representation space and combines it with a cross-modal attention mechanism to achieve effective alignment between sign language action sequences and natural language semantics. Compared with traditional direct mapping methods, it can establish more stable semantic relationships between different modalities. By introducing structured semantic intermediate representations in the sign language recognition and generation process, semantic information is uniformly expressed during bidirectional conversion, thereby improving the accuracy of sign language to text / speech conversion and the consistency of text / speech to sign language generation. Especially in complex sentence structures or continuous action expression scenarios, it can significantly reduce semantic bias and information loss.

[0016] 2. This invention achieves effective modeling of dynamic features of sign language by standardizing the sequence of key points and combining temporal coding with an action generation model. Simultaneously, through a collaborative processing mechanism between the edge and cloud, it ensures both recognition accuracy and real-time performance while maintaining computational efficiency. Furthermore, by introducing smoothness constraints and motion consistency constraints, the continuity and naturalness of generated sign language actions are improved, making the output more consistent with actual sign language expression habits. This enhances the system's practicality and scalability in applications such as intelligent interaction and barrier-free communication. Attached Figure Description

[0017] Figure 1 This document presents exemplary flowcharts of deep learning-based sign language and speech translation methods and systems, as illustrated in some embodiments of this specification. Detailed Implementation

[0018] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the linguistic context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0019] It should be understood that the terms "system," "device," "unit," and / or "module" as used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0020] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0021] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0022] The following describes in detail, with reference to the accompanying drawings, the deep learning-based sign language and speech translation method and system provided in the embodiments of this specification.

[0023] Figure 1 This is an exemplary flowchart illustrating a deep learning-based sign language speech translation method and system according to some embodiments of this specification. In some embodiments, the deep learning-based sign language speech translation method and system can be executed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (instructions running on a processing device to execute hardware simulations), and any combination thereof. In some embodiments, Figure 1 One or more operations in the flowchart of the deep learning-based sign language speech translation method and system shown can be implemented by a processing device and / or a terminal device. For example, the deep learning-based sign language speech translation method and system can be stored in a storage device in the form of computer programs and / or instructions, and can be invoked and / or executed by the processing device and / or the terminal device.

[0024] The implementation process of each functional module will be further described in detail below, in conjunction with the system structure and data processing flow of the present invention. It should be understood that the content described in this section is a specific elaboration of the technical solutions in the claims, intended to enable those skilled in the art to implement the present invention accordingly.

[0025] In practical engineering implementation, the functional modules described in this invention can run as software programs on servers, personal computers, mobile terminals, or embedded devices, and can invoke GPUs, NPUs, or other dedicated hardware acceleration units to execute them according to computing needs. Data interaction between modules can be achieved through shared memory, function interface calls, or network communication, such as through inter-process communication mechanisms or data transmission based on protocols such as HTTP and WebSocket. The overall system can be deployed using a modular or distributed architecture to balance real-time performance and computational efficiency.

[0026] It should be noted that this invention is not limited to a specific deployment form and can be implemented in centralized processing or edge-cloud collaborative architecture.

[0027] The technical solution of the present invention will be further described below with reference to specific embodiments.

[0028] (I) Multimodal data acquisition and synchronization In this embodiment, the multimodal acquisition module is used to acquire sign language video data, voice data, and text data, and to provide basic input for subsequent cross-modal processing.

[0029] Specifically, sign language video data can be captured using an RGB camera with a resolution of 640×480, 1280×720, or higher, and the frame rate can be set to 15fps, 25fps, or 30fps depending on application requirements. In some implementations, depth cameras can also be introduced to obtain three-dimensional spatial information, thereby improving the accuracy of subsequent motion modeling.

[0030] Voice data is collected via a microphone, with a sampling rate of 16kHz or higher; text data can be obtained through a user input interface or an external interface.

[0031] To avoid time misalignment between different modalities when acquiring multimodal data simultaneously, this implementation introduces a unified time reference. Specifically, a timestamp can be added to each frame of video data and its corresponding audio segment, and synchronization can be achieved through time window alignment or buffering mechanisms. For cases with inconsistent sampling frequencies, time alignment can be achieved through interpolation or resampling, thereby ensuring consistency of data across different modalities in the time dimension.

[0032] The above method provides temporally consistent input data for subsequent cross-modal semantic mapping, which plays an important role in improving semantic alignment accuracy.

[0033] (II) Key Sign Language Points Extraction and Preprocessing The visual feature extraction module is used to extract key hand information from sign language videos and generate a structured sequence of movements.

[0034] First, the input video is decoded frame by frame, and the images are preprocessed, including scaling, normalization, and color space conversion, to improve the stability of the model processing.

[0035] Subsequently, the hand region is located using an object detection model. This detection model can be a single-stage or two-stage detection structure based on a convolutional neural network, used to output the bounding box coordinates of the hand region. Compared to directly detecting key points on the entire image, performing region localization first helps reduce interference and improve key point detection accuracy.

[0036] After obtaining the hand region, keypoints are extracted using a keypoint detection model. Preferably, 21 keypoints are extracted for each hand, resulting in 42 keypoints per frame. Keypoint coordinates can be represented as two-dimensional coordinates (x, y), or extended to three-dimensional coordinates (x, y, z) when depth information is available.

[0037] For consecutive video frames, key points are arranged in chronological order to form a key point temporal data sequence. To improve the consistency and modelability of the sequence, this embodiment further processes the sequence, including: The coordinates of key points are normalized to eliminate the effects of different shooting scales; Missing key points are filled by interpolation to avoid sequence interruption; Filtering methods are used to smooth the sequence in order to reduce detection noise; Constrain or correct outliers; Sequences of different lengths are processed uniformly by resampling or truncation.

[0038] After the above processing, a standardized sign language key point action sequence is obtained, which can be represented as a tensor in the form of T×N×D, where T is the time length, N is the number of key points, and D is the coordinate dimension.

[0039] It should be noted that by standardizing the key point sequence, the impact of individual differences and differences in collection conditions on the model can be effectively reduced, thereby improving the stability of subsequent semantic modeling.

[0040] (III) Sign Language Timing Coding The sign language temporal coding module is used to extract features in the temporal dimension from key point action sequences.

[0041] In this embodiment, a Transformer coding network can be used as the main temporal modeling structure. First, the keypoint coordinates are mapped to a high-dimensional feature space through linear mapping, and positional encoding is introduced to preserve temporal order information.

[0042] The encoding network consists of multiple layers of encoding units, each including a multi-head self-attention mechanism and a feedforward neural network structure, combined with residual connections and normalization operations. Through the multi-head attention mechanism, both short-term dynamic changes and long-term dependencies can be modeled simultaneously, thereby improving the expressive ability of complex sign language movements.

[0043] In other implementations, bidirectional long short-term memory networks, gated recurrent unit networks, or temporal convolutional networks may be used as alternative or supplementary structures.

[0044] After encoding, a temporal encoded feature sequence containing contextual information is obtained.

[0045] (iv) Cross-modal semantic mapping The cross-modal semantic mapping module is used to map sign language temporal features to a unified semantic representation space, thereby achieving semantic alignment between sign language and text.

[0046] In this embodiment, a unified semantic representation space is constructed, which adopts vector embedding. Data from different modalities can be mapped to this space for unified expression. Unlike direct end-to-end mapping, this invention decouples sign language actions from linguistic semantics by introducing a unified semantic representation layer, thereby improving the system's generalization ability.

[0047] Specifically, a cross-modal attention mechanism can be used to implement the mapping process, using temporal encoded features as query vectors and semantic embedding representations as keys and values, and obtaining alignment results through attention calculation. Simultaneously, different time segments are weighted according to attention weights to highlight the contribution of key actions to semantic expression.

[0048] After mapping, a structured intermediate representation of sign language semantics is obtained. This representation not only contains semantic vectors, but can also explicitly represent sign language term boundaries, word order relationships, or semantic role information.

[0049] It is important to emphasize that by introducing a structured semantic intermediate representation, subsequent text generation and action generation can be based on the same semantic foundation, thus achieving consistency in the bidirectional conversion process. This structure differs from traditional direct mapping methods and exhibits better stability and scalability in multimodal translation tasks.

[0050] (v) Natural Language Generation The natural language generation module is used to generate target text sequences based on semantic intermediate representations.

[0051] In this implementation, a generative model based on the Transformer architecture can be used. This model generates text step by step through an autoregressive approach based on the semantic representation of the input. During the generation process, the output at each step depends not only on the semantic representation but also on the previously generated historical sequence, thereby ensuring the coherence of the sentences.

[0052] Furthermore, the generated results can be optimized using strategies such as bundle search. In some implementations, post-processing of the generated text, such as grammar correction or punctuation completion, can further improve readability.

[0053] (vi) Speech processing The speech processing module is used to convert between text and speech.

[0054] In the process of text-to-speech, a neural network speech synthesis model can be used to convert the text sequence into a speech signal output; in the process of speech-to-text, a speech recognition model can be used to convert the input speech into text.

[0055] The converted text can be used as input for semantic encoding, thereby enabling indirect conversion between speech and sign language.

[0056] (vii) Generation of sign language gestures The sign language action generation module is used to generate sign language action sequences based on text or semantic representations.

[0057] First, the input text is encoded using a text semantic encoding unit and mapped to a unified semantic representation space. Since this semantic space is consistent with the sign language semantic representation space, it can be directly used as input for action generation.

[0058] Subsequently, the action sequence generation unit generates a sign language keypoint sequence based on the semantic representation. In one implementation, an autoregressive model can be used to progressively predict the keypoint coordinates at each time step; in another implementation, a diffusion generation model can be used to obtain the complete action sequence through progressive generation.

[0059] To ensure the continuity and rationality of the generated actions, smoothness constraints or motion consistency constraints can be introduced during the generation process to avoid discontinuous or unnatural action changes.

[0060] It should be noted that since action generation and semantic representation share a unified representation space, the generated results are consistent with the input text at the semantic level, thereby improving the accuracy of bidirectional translation.

[0061] (viii) Interactive Output and System Integration The interactive output module is used to output system results in a unified manner.

[0062] Specifically, this includes: displaying the text results on the terminal device; outputting the voice signal to the speaker; and inputting the key point sequence into the rendering module to drive the virtual character to perform sign language actions. In some implementations, a skeleton-driven 3D model can be used for visualization.

[0063] In terms of system integration, modules can use a unified data structure for transmission, such as tensor or sequence structure, to ensure interface consistency.

[0064] (ix) Collaborative implementation between the device and the cloud In another embodiment, the present invention may employ a collaborative processing architecture between the edge and the cloud.

[0065] For example, video acquisition, preprocessing, and key point extraction can be performed on the edge device to reduce data transmission pressure, while computationally complex time-series coding, semantic mapping, and generation modules can be deployed in the cloud.

[0066] Data exchange between the edge and the cloud is achieved via the network, thereby optimizing the allocation of computing resources. In situations where network conditions are limited, a simplified model can also be deployed on the edge to support basic functionalities.

[0067] Through the collaborative processing of the above modules, bidirectional translation between sign language and speech or text is achieved. This invention, by introducing a unified semantic representation space and a cross-modal attention mechanism, makes the semantic alignment process between different modalities more stable, thus achieving good translation results even in complex sign language scenarios, and has strong practical value.

[0068] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A deep learning-based sign language-speech translation system, characterized in that, include: The multimodal acquisition module is used to acquire sign language video data, voice data, and text data; The visual feature extraction module is used to detect and track the hand area in the sign language video data, extract the two-dimensional or three-dimensional key point coordinate sequence of the hands in continuous video frames, and perform noise reduction, smoothing and time sequence alignment processing on the key point coordinate sequence to generate a standardized sign language key point action sequence. The sign language temporal coding module is used to input the standardized sign language key point action sequence into a temporal coding network containing a multi-head attention mechanism to extract the local dynamic features and global dependency features of the sign language actions in the time dimension and output the temporal coding features. A cross-modal semantic mapping module is used to map the temporal encoded features to a unified semantic representation space through a cross-modal attention mechanism. The unified semantic representation space is a semantic representation space based on vector embedding, and generates a structured sign language semantic intermediate representation. The structured sign language semantic intermediate representation includes at least one of the following information: sign language word sequence, action boundary marker, semantic role relationship, and word order dependency relationship. The natural language generation module is used to input the structured sign language semantic intermediate representation into a compressed and optimized neural network language model, reconstruct the word order, perform semantic completion and syntactic optimization, and generate the target text sequence. The speech processing module is used to realize bidirectional conversion between text data and speech data, including: converting the target text sequence into speech output, and converting the input speech into text to be parsed; The sign language action generation module includes: a text semantic encoding unit, used to encode the text to be parsed into a semantic feature vector and map it to the unified semantic representation space, wherein the text to be parsed includes text obtained by speech conversion or collected text data; An action sequence generation unit is used to generate a sign language key point action sequence or a sign language skeleton driven sequence through a sequence generation network based on the semantic representation in the unified semantic representation space. The interactive output module is used to output the target text sequence, speech output, and sign language visualization results driven by the sign language key point action sequence, so as to realize bidirectional translation between sign language and speech / text.

2. The system according to claim 1, characterized in that: The visual feature extraction module also includes a temporal normalization processing unit, which is used to align the temporal scales of sign language key point action sequences of different lengths and unify the sequence length through interpolation or resampling to improve the stability and robustness of subsequent temporal modeling.

3. The system according to claim 1, characterized in that: The temporal coding network is a combination of at least one of the following structures: bidirectional long short-term memory network, gated recurrent unit network, temporal convolutional network, and Transformer coding network, and enhances the ability to model long-term dependencies through a multi-layer stacked structure.

4. The system according to claim 1, characterized in that: The cross-modal semantic mapping module adopts a multi-head cross-modal attention mechanism, realizes semantic alignment between sign language and text through a unified semantic representation space, and enhances the representation of key action segments through attention weights to improve the accuracy of semantic mapping.

5. The system according to claim 1, characterized in that: The neural network language model is a lightweight model that has undergone quantization, pruning, or knowledge distillation, and supports low-latency inference execution on resource-constrained devices.

6. The system according to claim 1, characterized in that: The action sequence generation unit is based on an autoregressive sequence generation model or a diffusion generation model. It maps semantic feature vectors into a continuous sequence of sign language key points and optimizes the continuity of the generated actions through smoothness constraints or motion consistency constraints.

7. A deep learning-based sign language-to-speech translation method, characterized in that, Includes the following steps: S1: Collect sign language video data, voice data, and text data; S2: Perform hand detection, tracking and key point extraction on the sign language video data to obtain temporal data of hand key points, and perform noise reduction, smoothing and temporal alignment processing to generate a sequence of sign language key point actions; S3: Input the sign language key point action sequence into a temporal coding network containing a multi-head attention mechanism for encoding to obtain temporal coding features; S4: The temporal coding features are mapped to a unified semantic representation space through a cross-modal attention mechanism to generate a structured sign language semantic intermediate representation; S5: Input the structured sign language semantic intermediate representation into the neural network language model to generate the target text sequence, and convert the target text sequence into speech output; S6: Convert the input speech into text to be parsed, or acquire text data, and encode it into the unified semantic representation space through a text semantic coding model; S7: Generate a sequence of sign language key points based on the semantic representation in the unified semantic representation space, and output the corresponding sign language visualization results, thereby realizing bidirectional translation between sign language and speech / text.

8. The method according to claim 7, characterized in that, Also includes: Based on user geographic information, historical interaction data, and user feedback information, the temporal coding network, cross-modal semantic mapping model, and action generation model are adaptively updated to achieve dynamic adaptation to different sign language expressions. as well as The preprocessing of sign language video data, key point extraction, and some reasoning processes are deployed on the edge device to reduce the system's dependence on the network environment and to achieve basic translation functions under offline conditions.

Images