Audio-to-sign language conversion method and device, equipment, readable storage medium and program product

Abstract

The invention relates to an audio-to-sign language conversion method and device, equipment, a readable storage medium and a program product. The method comprises the following steps: capturing real-time audio on a webpage; performing voice recognition on the real-time audio by adopting a pre-trained audio recognition model to obtain a text sequence corresponding to the real-time audio; analyzing the text sequence to obtain a plurality of sign language actions; and synthesizing the plurality of sign language actions into a continuous sign language animation, displaying the sign language animation in the webpage, and playing real-time audio. By adopting the method, the real-time performance of converting the audio into the sign language can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio-to-sign language method, device, equipment, readable storage medium, and program product. Background Art

[0002] In order to enhance the accessibility of the Internet and enable hearing-impaired people to better understand and receive information when browsing websites containing audio content, the traditional method is to process real-time audio to sign language in the server. However, this method has problems such as high latency and high resource consumption, especially in real-time interactive scenarios. In addition, the generation of sign language animations is often limited to offline processing and cannot meet real-time requirements. Summary of the invention

[0003] Based on this, it is necessary to provide an audio-to-sign language method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the real-time performance of audio-to-sign language conversion in response to the above technical problems.

[0004] In a first aspect, the present application provides a method for converting audio to sign language, comprising:

[0005] Capture live audio on web pages;

[0006] Use a pre-trained audio recognition model to perform speech recognition on real-time audio to obtain a text sequence corresponding to the real-time audio;

[0007] Parse the text sequence to obtain multiple sign language actions;

[0008] Combine multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play real-time audio.

[0009] In one embodiment, a pre-trained audio recognition model is used to perform speech recognition on real-time audio to obtain a text sequence corresponding to the real-time audio, including:

[0010] Preprocessing of real-time audio, including noise suppression, automatic gain control and echo cancellation;

[0011] The pre-trained audio recognition model is used to perform speech recognition on the pre-processed audio to obtain the text sequence corresponding to the real-time audio.

[0012] In one embodiment, a pre-trained audio recognition model is used to perform speech recognition on the pre-processed audio to obtain a text sequence corresponding to the real-time audio, including:

[0013] Perform feature extraction on the preprocessed audio to obtain a feature vector that can be understood by the audio recognition model;

[0014] The pre-trained audio recognition model is used to perform speech recognition on the feature vector to obtain the text sequence corresponding to the real-time audio.

[0015] In one embodiment, the text sequence is parsed to obtain a plurality of sign language actions, including:

[0016] Perform text preprocessing on the text sequence to obtain multiple processed text words. Text preprocessing includes text cleaning, lexical analysis, grammatical analysis, and named entity recognition.

[0017] Convert each processed text word into a corresponding semantic vector;

[0018] Convert multiple semantic vectors into action sequences through a pre-trained sequence-to-sequence model;

[0019] Based on the action sequence, a plurality of sign language actions are determined.

[0020] In one embodiment, based on the action sequence, a plurality of sign language actions are determined, including:

[0021] Obtaining a preset sign language vocabulary library, wherein the preset sign language vocabulary library stores the correspondence between sign language vocabulary and sign language actions;

[0022] Based on a preset sign language vocabulary library, multiple actions with a combination relationship in an action sequence are mapped to corresponding sign language actions to obtain multiple sign language actions.

[0023] In one embodiment, the audio to sign language conversion method further includes:

[0024] Render the sign language animation and display the rendered sign language animation on the web page.

[0025] In a second aspect, the present application also provides an audio-to-sign language device, comprising:

[0026] Capture module, used to capture real-time audio on web pages;

[0027] A recognition module is used to perform speech recognition on real-time audio using a pre-trained audio recognition model to obtain a text sequence corresponding to the real-time audio;

[0028] A parsing module, used to parse the text sequence to obtain multiple sign language actions;

[0029] The display module is used to synthesize multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play real-time audio.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Capture live audio on web pages;

[0032] Use a pre-trained audio recognition model to perform speech recognition on real-time audio to obtain a text sequence corresponding to the real-time audio;

[0033] Parse the text sequence to obtain multiple sign language actions;

[0034] Combine multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play real-time audio.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0036] Capture live audio on web pages;

[0037] Use a pre-trained audio recognition model to perform speech recognition on real-time audio to obtain a text sequence corresponding to the real-time audio;

[0038] Parse the text sequence to obtain multiple sign language actions;

[0039] Combine multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play real-time audio.

[0040] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0041] Capture live audio on web pages;

[0042] Use a pre-trained audio recognition model to perform speech recognition on real-time audio to obtain a text sequence corresponding to the real-time audio;

[0043] Parse the text sequence to obtain multiple sign language actions;

[0044] Combine multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play real-time audio.

[0045] The above-mentioned audio-to-sign language method, device, computer equipment, computer-readable storage medium and computer program product capture the real-time audio on the web page; use a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; parse the text sequence to obtain multiple sign language actions; synthesize the multiple sign language actions into a continuous sign language animation, display the sign language animation on the web page and play the real-time audio. In the above scheme, since the speech recognition of the real-time audio, the text sequence parsing, and the synthesis of the sign language animation are all performed in the terminal where the web page is located, the delay problem caused by processing in the server is avoided, which is conducive to improving the real-time performance of audio-to-sign language conversion; at the same time, the pre-trained audio recognition model is a deep learning model, and the introduction of the deep learning model for speech recognition can significantly improve the efficiency of speech recognition, further improving the real-time performance of audio-to-sign language conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A diagram showing an application environment of an audio-to-sign language method in an embodiment;

[0048] Figure 2 A schematic diagram of a flow chart of an audio to sign language conversion method in one embodiment;

[0049] Figure 3 A flowchart of a method for speech processing in one embodiment;

[0050] Figure 4 A schematic diagram of a method flow for performing speech recognition using an audio recognition model in one embodiment;

[0051] Figure 5 A schematic diagram of a method flow for parsing a text sequence in one embodiment;

[0052] Figure 6 A schematic diagram of a method for generating a sign language animation in one embodiment;

[0053] Figure 7 Schematic diagram of the overall process of an audio to sign language conversion method in one embodiment;

[0054] Figure 8 is a structural block diagram of an audio to sign language device in one embodiment;

[0055] Fig. 9FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The audio to sign language conversion method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 has a web page, and the terminal 102 captures the real-time audio on the web page; uses a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; parses the text sequence to obtain multiple sign language actions; synthesizes multiple sign language actions into a continuous sign language animation, displays the sign language animation on the web page and plays the real-time audio. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0058] In an exemplary embodiment, Figure 2 As shown, a method for converting audio to sign language is provided, and the method is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, including the following steps 202 to 208. Among them:

[0059] Step 202, capturing real-time audio on the web page.

[0060] The web page refers to a web page in a web application, for example, a web page in a browser. The real-time audio refers to an audio stream played in a web page, which may be audio data or audio data in a video.

[0061] In some embodiments, real-time audio on a web page can be captured using technologies such as Web Audio API (Web Audio Processing Interface) or WebRTC (Web Real-Time Communications). These technologies allow real-time processing of audio data in a web page without sending the data to a server, thereby reducing latency, improving audio processing efficiency, and protecting user privacy. For example, WebRTC's RTCPeerConnection and getUserMediaAPI can be used to ensure smooth acquisition of microphone audio streams.

[0062] In some embodiments, the audio-to-sign language method proposed in the embodiments of the present application runs in a WebAssembly environment. WebAssembly is an efficient, low-level bytecode format that can be used for direct execution in modern browsers. Applying it to the real-time audio-to-sign language method can bring the following benefits: 1. Performance improvement: WebAssembly is closer to native machine code than JavaScript and has higher execution efficiency. This can significantly improve the speed of speech recognition and animation rendering, reduce latency, and make real-time processing smoother. 2. Cross-platform compatibility: WebAssembly is a standard that is supported by all modern browsers. This means that code written in WebAssembly can run consistently in different browsers, solving the portability challenges of traditional methods. 3. Security: WebAssembly runs in the browser's sandbox environment, which allows efficient code to be safely executed without sacrificing performance, and has high security.

[0063] Since web pages usually run in a JavaScript environment, data captured from the web pages can be passed to the WebAssembly environment. For example, real-time audio is directly imported into the WebAssembly environment, avoiding unnecessary data copying and format conversion, and significantly improving processing efficiency.

[0064] The TypedArray and SharedArrayBuffer mechanisms can be used to transfer data between WebAssembly and JavaScript environments to ensure efficient data exchange and thread safety, further reducing processing delays. At the same time, through WebAssembly's Atomics API and Mutex mechanism, data access coordination in a multi-threaded environment is achieved, ensuring the stable operation of the system.

[0065] Step 204: Use a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio.

[0066] The audio recognition model is a pre-trained deep neural network (DNN) or convolutional neural network (CNN). Using DNN or CNN technology to perform speech recognition on real-time audio can significantly improve the accuracy of speech recognition, and can better understand and process natural language text, and improve the accuracy and fluency of sign language conversion.

[0067] In some embodiments, a pre-trained audio recognition model can be loaded and initialized in a WebAssembly environment. The audio recognition model is converted into a WebAssembly module using tools such as TensorFlow.js, and the high performance characteristics of WebAssembly are utilized to significantly improve the execution efficiency of the model. The initialization step includes allocating necessary memory and resources, creating the network structure of the model, and loading weight parameters.

[0068] Step 206: parse the text sequence to obtain multiple sign language actions.

[0069] Among them, sign language movements refer to the use of gestures to measure movements, and simulate images or syllables according to the changes in gestures to form certain meanings or words. It is a language of hands for people with hearing impairments or who cannot speak to communicate with each other and exchange ideas. In some embodiments, deep learning models and natural language processing (NLP) technology can be used to parse text sequences in a WebAssembly environment and convert them into sign language movements. Traditional sign language conversion systems usually rely on predefined sign language dictionaries and lack flexibility and adaptability. Using deep learning and NLP technology, the input text sequence can be parsed more accurately and the corresponding sign language movements can be generated, thereby improving the accuracy and naturalness of the conversion.

[0070] Step 208, synthesize the multiple sign language actions into a continuous sign language animation, display the sign language animation on the web page and play the real-time audio.

[0071] In order to ensure the coherence and naturalness of the sign language movements displayed on the web page and avoid unnatural transitions or repetitions, the sign language movements can be synthesized into continuous sign language animations in real time. For example, in the WebAssembly environment, multiple sign language movements are synthesized into continuous sign language animations through an animation engine, and optimized to ensure fluency and naturalness.

[0072] In the above-mentioned audio-to-sign language method, real-time audio on a web page is captured; a pre-trained audio recognition model is used to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; the text sequence is parsed to obtain multiple sign language actions; the multiple sign language actions are synthesized into a continuous sign language animation, and the sign language animation is displayed on the web page and the real-time audio is played. In the above-mentioned scheme, since the speech recognition of real-time audio, text sequence parsing, and sign language animation synthesis are all performed on the web page, the delay problem caused by processing in the server is avoided, which is conducive to improving the real-time performance of audio-to-sign language conversion; at the same time, the pre-trained audio recognition model is a deep learning model, and the introduction of a deep learning model for speech recognition can significantly improve the efficiency of speech recognition, further improving the real-time performance of audio-to-sign language conversion.

[0073] In an exemplary embodiment, a pre-trained audio recognition model is used to perform speech recognition on real-time audio to obtain a text sequence corresponding to the real-time audio, including: preprocessing the real-time audio, the preprocessing including noise suppression, automatic gain control and echo cancellation; and using a pre-trained audio recognition model to perform speech recognition on the preprocessed audio to obtain a text sequence corresponding to the real-time audio.

[0074] Among them, the captured real-time audio often contains more noise. The real-time audio can be processed through preprocessing to reduce background noise and standardize the audio format to provide clean and consistent input for the subsequent deep learning model's speech recognition, so as to improve the accuracy of speech recognition.

[0075] Preprocessing methods include but are not limited to noise suppression, automatic gain control, and echo cancellation. Among them, noise suppression and automatic gain control algorithms effectively improve audio quality, laying the foundation for accurate recognition of subsequent deep learning models; echo cancellation algorithms specifically solve the common echo problem in duplex communication, greatly improving user experience.

[0076] In this embodiment, by performing speech recognition after real-time preprocessing, noise interference in the audio data is effectively suppressed, the audio quality is improved, and the accuracy of speech recognition is improved.

[0077] In an exemplary embodiment, a pre-trained audio recognition model is used to perform speech recognition on pre-processed audio to obtain a text sequence corresponding to the real-time audio, including: extracting features from the pre-processed audio to obtain a feature vector that can be understood by the audio recognition model; and performing speech recognition on the feature vector using a pre-trained audio recognition model to obtain a text sequence corresponding to the real-time audio.

[0078] Among them, Figure 3The figure shows a flow chart of a method for speech processing in one embodiment. After the real-time audio is imported into the WebAssembly environment, it is preprocessed in the WebAssembly environment. The preprocessed audio needs to be feature extracted to obtain a feature vector that can be understood by the audio recognition model before it can be input into the pre-trained audio recognition model for speech recognition. For example, such as Mel-frequency cepstral coefficients (MFCC) or filter group energy, this process is crucial for audio recognition and analysis. Through the efficient execution of WebAssembly, feature extraction is not only fast, but also highly accurate, providing a high-quality data foundation for subsequent deep learning model input.

[0079] like Figure 4 The figure shows a flow chart of a method for speech recognition by an audio recognition model in one embodiment. The audio recognition model converts real-time audio into a corresponding text sequence through forward propagation calculations of a multi-layer neural network. In this process, a convolutional neural network (CNN) is first used to extract high-order features. CNN processes the input features layer by layer through multiple convolutional layers and pooling layers to extract more abstract feature representations. Next, a recursive neural network (RNN) or a long short-term memory network (LSTM) processes time series data to capture the temporal dependencies in real-time audio. The RNN or LSTM processes the feature vectors step by step through multiple hidden layers to generate hidden states for each time step. Finally, the output of the RNN or LSTM is connected to a fully connected layer through a fully connected layer to generate a class probability distribution for each time step, and the output is converted into a probability distribution using a softmax activation function.

[0080] In order to improve the accuracy of recognition, the embodiment of the present application adopts the connection temporal classification (CTC) decoding algorithm to decode the most likely text sequence from the output probability distribution of the model. CTC aligns the input features and the target text sequence by adding blank symbols, and handles inputs and outputs of different lengths. During the decoding process, greedy search and beam search algorithms are used to find the most likely character sequence from the output probability distribution. Greedy search selects the character with the highest probability at each time step, and beam search improves recognition accuracy by maintaining a set of candidate sequences and comprehensively considering multiple possibilities. Finally, consecutive repeated characters are merged and blank symbols are removed to obtain the final text output.

[0081] In this embodiment, the pre-processed audio is converted into a feature vector, and then input into a pre-trained audio recognition model to perform speech recognition on the feature vector, which is helpful to improve the accuracy of speech recognition.

[0082] In an exemplary embodiment, parsing a text sequence to obtain a plurality of sign language actions includes: performing text preprocessing on the text sequence to obtain a plurality of processed text words, the text preprocessing including text cleaning, lexical analysis, grammatical analysis, and named entity recognition; converting each processed text word into a corresponding semantic vector; converting the plurality of semantic vectors into action sequences through a pre-trained sequence-to-sequence model; and determining a plurality of sign language actions based on the action sequences.

[0083] Among them, natural language processing technology can be used to preprocess the text sequence. Text preprocessing includes but is not limited to text cleaning, lexical analysis, grammatical analysis, named entity recognition, etc. Among them, text cleaning is used to remove irrelevant characters in the text sequence, such as punctuation marks, special characters, etc., to ensure the purity of the input. Lexical analysis is used to perform operations such as word segmentation and part-of-speech tagging on the text sequence to understand the role of each word in the sentence. Syntactic analysis is used to build a syntax tree, understand the sentence structure, and provide a basis for subsequent semantic analysis. Named entity recognition is used to identify proper nouns in text sequences, such as names of people and places, which may require specific gestures to express.

[0084] The processed text words can be converted into corresponding multiple sign language actions through a deep learning model. Each processed text word needs to be converted into a corresponding semantic vector, which helps capture word meaning and context information.

[0085] In some embodiments, a sequence-to-sequence (Seq2Seq) model can be used to convert text sequences into action sequences using the Seq2Seq architecture combined with an attention mechanism. During the training of the sequence-to-sequence model, the training dataset should contain a large amount of text and corresponding sign language videos or action annotations for model training. In some embodiments, a gesture prediction model can also be used to predict the specific details of each gesture, such as hand shape, position, and motion trajectory.

[0086] like Figure 5 The figure shows a flow chart of a method for parsing a text sequence in one embodiment. An action sequence is often composed of actions corresponding to a single word or character. Directly outputting the action sequence may lead to a poor user experience. Therefore, the action sequence can be converted into a coherent sign language action to facilitate user understanding.

[0087] In this embodiment, by performing text preprocessing on the text sequence, irrelevant characters in the text sequence can be reduced, and the text sequence can be converted into easy-to-understand text words, thereby improving the recognition accuracy of sign language movements; the method of using a sequence-to-sequence model to convert the semantic vectors corresponding to the text words into action sequences, and then converting them into sign language movements, further improves the conversion efficiency and expression effect of sign language movements.

[0088] In an exemplary embodiment, multiple sign language actions are determined based on an action sequence, including: obtaining a preset sign language vocabulary library, which stores the correspondence between sign language vocabulary and sign language actions; based on the preset sign language vocabulary library, mapping multiple actions with a combination relationship in the action sequence to corresponding sign language actions to obtain multiple sign language actions.

[0089] Among them, the sign language vocabulary includes a single word, a character or a text word composed of multiple words. There is a corresponding sign language action for each sign language vocabulary. Since the text word is composed of multiple words, each word corresponds to an action, so the multiple actions with a combination relationship are the actions corresponding to the multiple words that can be combined into the text word. The multiple actions with a combination relationship are mapped to the corresponding sign language actions. For example, the actions corresponding to "hand" and "language" can be mapped to the sign language actions corresponding to "sign language".

[0090] In this embodiment, by searching the corresponding relationship between each sign language vocabulary and the sign language action in the preset sign language vocabulary library, multiple actions with a combination relationship are mapped to corresponding sign language actions, which is beneficial to improving the coherence and expression effect of the sign language actions.

[0091] In an exemplary embodiment, the audio-to-sign language method further includes: rendering a sign language animation, and displaying the rendered sign language animation on a web page.

[0092] The rendering of the sign language animation can be done by WebGL rendering. For example, the sign language animation is sent to the WebGL environment and rendered into three-dimensional graphics in real time to create high-quality sign language animation visual effects. WebGL uses hardware acceleration to ensure that the sign language animation can run smoothly on different devices.

[0093] like Figure 6 The figure is a flow chart of a method for generating a sign language animation in one embodiment. After a plurality of sign language actions are generated, the plurality of sign language actions are optimized and synthesized into a sign language animation in real time, the sign language animation is loaded into a WebAssembly environment, rendered through WebGL, and the rendered sign language animation is outputted in a web page.

[0094] The rendered sign language animation is displayed directly in the user's browser window (web page) without the need for additional plug-ins or software.

[0095] In this embodiment, using WebGL for rendering can not only improve the quality and performance of the animation, but also make full use of the graphics processing capabilities of modern browsers and devices to provide users with an immersive and responsive experience. This technology selection is particularly suitable for applications that require high visualization and interactivity, such as sign language animation demonstrations.

[0096] To explain the audio-to-sign language method and its effect in detail, a most detailed embodiment is described below:

[0097] This article takes the audio-to-sign language conversion scenario in a web browser as an example. Figure 7 The figure is a schematic diagram of the overall flow of an audio to sign language method in one embodiment.

[0098] The audio-to-sign language method includes audio capture and preprocessing, speech recognition using deep learning, text parsing and sign language conversion using deep learning, and WebGL sign language animation generation.

[0099] (1) Audio capture and preprocessing

[0100] WebRTC audio capture and preprocessing are the foundation of real-time audio communication, which ensures high-quality audio data transmission. The preprocessing step optimizes audio quality, reduces background noise, and standardizes the audio format to provide clean and consistent input for subsequent deep learning model speech recognition.

[0101] In the audio capture phase, WebRTC's RTCPeerConnection and getUserMedia API are used to ensure smooth acquisition of microphone audio streams. Different from the traditional JavaScript processing flow, the embodiment of the present application directly imports real-time audio into the WebAssembly environment, avoiding unnecessary data copying and format conversion, and significantly improving processing efficiency.

[0102] Inside WebAssembly, audio data flows through a series of carefully designed preprocessing algorithms, including but not limited to key steps such as noise suppression, automatic gain control, echo cancellation, and feature extraction. These algorithms can be written in C / C++ and compiled into WebAssembly modules through the Emscripten toolchain, ensuring the ultimate performance of algorithm execution.

[0103] In the feature extraction phase, the WebAssembly module converts real-time audio into feature vectors that can be understood by the deep learning model. Through the efficient execution of WebAssembly, feature extraction is not only fast but also highly accurate, providing a high-quality data foundation for subsequent deep learning model input.

[0104] The data transmission between WebAssembly and JavaScript environment adopts TypedArray and SharedArrayBuffer mechanism to ensure the efficiency and thread safety of data exchange, and further reduce the processing delay. At the same time, through WebAssembly's Atomics API and Mutex mechanism, data access coordination in a multi-threaded environment is achieved, ensuring the stable operation of the system.

[0105] (2) Introducing deep learning speech recognition

[0106] The captured audio stream goes through a series of preprocessing steps and extracts Mel-frequency cepstral coefficients (MFCC) to meet the input requirements of the deep learning model. The pre-trained deep learning model is loaded and initialized in the WebAssembly environment. The deep learning model is converted into a WebAssembly module using tools such as TensorFlow.js, and the high-performance characteristics of WebAssembly are used to significantly improve the execution efficiency of the model. The initialization step includes allocating the necessary memory and resources, creating the network structure of the model, and loading the weight parameters.

[0107] The preprocessed MFCC feature vector is input into the deep learning model for speech recognition.

[0108] (3) Introducing deep learning for text parsing and sign language conversion

[0109] Using deep learning models and natural language processing (NLP) technology, text is parsed and converted into sign language expressions in a WebAssembly environment. Traditional sign language conversion systems usually rely on predefined sign language dictionaries and lack flexibility and adaptability. Using deep learning and NLP technology can more accurately parse the input text and generate the corresponding sign language action sequence, thereby improving the accuracy and naturalness of the conversion.

[0110] The text preprocessing stage includes but is not limited to text cleaning, lexical analysis, grammatical analysis, named entity recognition, etc.

[0111] In deep learning model applications, semantic vector encoding uses pre-trained NLP models (such as BERT, GPT, etc.) to convert each word or phrase into a semantic vector, which helps capture word meaning and contextual information. Sequence-to-sequence (Seq2Seq) models use the Seq2Seq architecture combined with an attention mechanism to convert text sequences into sign language action sequences. The training dataset should contain a large amount of text and corresponding sign language videos or action sequence annotations for model training. Gesture prediction models may also require a separate model to predict the specific details of each gesture, such as hand shape, position, and motion trajectory.

[0112] Gesture library and mapping. Preset the sign language vocabulary library and build a database containing the correspondence between common sign language vocabulary and sign language movements, including basic gestures and combined gestures. Gesture mapping maps the action sequence output by the deep learning model to specific sign language movements. This step may require manual design of rules to make up for the deficiencies in model prediction.

[0113] WebAssembly environment deployment phase. Model loading: Packing the trained deep learning model into WebAssembly format for fast loading and running in the browser. Resource management: Managing the memory usage of WebAssembly modules to ensure the efficiency and stability of model operation. Performance optimization: Optimizing the model for the WebAssembly environment, including but not limited to model quantization, pruning and other technologies to improve the model's running speed and response time.

[0114] (4) WebGL sign language animation generation

[0115] Action sequence optimization, post-processing the generated gesture mapping sequence to ensure coherence and naturalness, and avoid unnatural transitions or repetitions. Real-time action synthesis, in the WebAssembly environment, through the animation engine, the action sequence is synthesized into a continuous sign language animation, and optimized to ensure fluency and naturalness.

[0116] WebGL rendering: animation data is fed into the WebGL environment and rendered into 3D graphics in real time in the browser, creating high-quality sign language animation visual effects. WebGL uses hardware acceleration to ensure that the animation runs smoothly on different devices. Output sign language animation: the rendered sign language animation is displayed directly in the user's browser window without the need for additional plug-ins or software.

[0117] Using WebGL for rendering not only improves the quality and performance of animations, but also makes full use of the graphics processing capabilities of modern browsers and devices to provide users with an immersive and responsive experience. This technology choice is particularly suitable for applications that require high visualization and interactivity, such as sign language animation demonstrations.

[0118] The above-mentioned audio-to-sign language method captures real-time audio on a web page; uses a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; parses the text sequence to obtain multiple sign language actions; synthesizes multiple sign language actions into a continuous sign language animation, displays the sign language animation on the web page and plays the real-time audio. In the above scheme, since the speech recognition of real-time audio, text sequence parsing, and sign language animation synthesis are all performed on the web page, the delay problem caused by processing in the server is avoided, which is conducive to improving the real-time performance of audio-to-sign language conversion; at the same time, the pre-trained audio recognition model is a deep learning model, and the introduction of a deep learning model for speech recognition can significantly improve the efficiency of speech recognition, further improving the real-time performance of audio-to-sign language conversion. In addition, the real-time audio-to-sign language animation conversion method that integrates Web technology through WebAssembly combines the emerging WebAssembly technology with multiple modules such as deep learning, audio capture, speech recognition, natural language processing, and sign language animation generation to form a complete real-time audio-to-sign language animation conversion system. The introduction of WebAssembly provides higher execution speed and better performance, making audio tracking smoother and more real-time. The system starts with audio signals and goes through a series of processes, including but not limited to audio capture, preprocessing, speech recognition and conversion into text, and then the text is converted into sign language movements after natural language processing analysis. Finally, WebGL technology is combined with deep learning models to generate corresponding sign language animations in real time. Advantages: Comprehensiveness. This system covers the entire process from audio input to sign language movement output, providing an end-to-end barrier-free communication solution. Real-time and fluency. Due to the close integration and optimization between modules, the entire process from audio reception to sign language animation generation can be carried out in real time, which improves the natural fluency of communication. User experience optimization provides intuitive and timely information feedback for hearing-impaired people, improves their communication experience in the network environment, and enhances participation and inclusiveness. Run deep learning models in the WebAssembly environment for speech recognition and natural language processing to achieve high performance and cross-platform compatibility. Advantages: Performance improvement. WebAssembly provides execution speed close to native code, making the operation of deep learning models more efficient. Cross-platform compatibility. The compatibility of WebAssembly on various operating systems and browsers ensures consistent performance of the model. Security is enhanced. Model and data processing are performed on the client, reducing the need for data transmission and enhancing privacy protection. Optimized audio capture and preprocessing. Audio streams are captured using WebRTC and optimized for preprocessing to meet the input requirements of deep learning models. Advantages: Reduced latency. Optimized preprocessing steps help reduce the latency from audio capture to speech recognition. Improved efficiency. Efficient preprocessing processes can speed up the processing of subsequent deep learning models and improve overall system performance.WebAssembly is the core of technical optimization and security enhancement. The binary format of WebAssembly can run indiscriminately on all supported platforms, ensuring that the real-time audio-to-sign language system can show consistent high performance regardless of the desktop or mobile terminal, regardless of the operating system. Advantages: Cross-platform consistency. No matter which device the user uses to access, they can enjoy the same level of service quality and performance, which greatly enhances the universal applicability and user satisfaction of the system. Safe and reliable. Data processing is completed locally, reducing dependence on third-party servers, enhancing the security and privacy protection of user data, and building user trust.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides an audio-to-sign language device for implementing the audio-to-sign language method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more audio-to-sign language device embodiments provided below can refer to the limitations of the audio-to-sign language method above, and will not be repeated here.

[0121] In an exemplary embodiment, Figure 8 As shown, an audio-to-sign language device 100 is provided, comprising: a capture module 120, a recognition module 140, a parsing module 160 and a display module 180, wherein:

[0122] A capture module 120, used to capture real-time audio on a web page;

[0123] The recognition module 140 is used to perform speech recognition on the real-time audio using a pre-trained audio recognition model to obtain a text sequence corresponding to the real-time audio;

[0124] A parsing module 160, used for parsing a text sequence to obtain a plurality of sign language actions;

[0125] The display module 180 is used to combine multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play real-time audio.

[0126] The above-mentioned audio-to-sign language device captures real-time audio on a web page; uses a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; parses the text sequence to obtain multiple sign language actions; synthesizes the multiple sign language actions into a continuous sign language animation, displays the sign language animation on the web page and plays the real-time audio. In the above scheme, since the speech recognition of real-time audio, text sequence parsing, and sign language animation synthesis are all performed on the web page, the delay problem caused by processing in the server is avoided, which is conducive to improving the real-time performance of audio-to-sign language conversion; at the same time, the pre-trained audio recognition model is a deep learning model, and the introduction of a deep learning model for speech recognition can significantly improve the efficiency of speech recognition, further improving the real-time performance of audio-to-sign language conversion.

[0127] In one embodiment, a pre-trained audio recognition model is used to perform speech recognition on real-time audio to obtain a text sequence corresponding to the real-time audio. The recognition module 140 is also used to: pre-process the real-time audio, the pre-processing includes noise suppression, automatic gain control and echo cancellation; use a pre-trained audio recognition model to perform speech recognition on the pre-processed audio to obtain a text sequence corresponding to the real-time audio.

[0128] In one embodiment, a pre-trained audio recognition model is used to perform speech recognition on the pre-processed audio to obtain a text sequence corresponding to the real-time audio. The recognition module 140 is also used to: extract features from the pre-processed audio to obtain a feature vector that can be understood by the audio recognition model; and use a pre-trained audio recognition model to perform speech recognition on the feature vector to obtain a text sequence corresponding to the real-time audio.

[0129] In one embodiment, a text sequence is parsed to obtain a plurality of sign language actions, and the parsing module 160 is further used to: perform text preprocessing on the text sequence to obtain a plurality of processed text words, the text preprocessing including text cleaning, lexical analysis, grammatical analysis, and named entity recognition; convert each processed text word into a corresponding semantic vector; convert the plurality of semantic vectors into action sequences through a pre-trained sequence-to-sequence model; and determine a plurality of sign language actions based on the action sequence.

[0130] In one embodiment, based on the action sequence, multiple sign language actions are determined, and the parsing module 160 is also used to: obtain a preset sign language vocabulary library, which stores the correspondence between sign language vocabulary and sign language actions; based on the preset sign language vocabulary library, map multiple actions with a combination relationship in the action sequence to corresponding sign language actions to obtain multiple sign language actions.

[0131] In one embodiment, the audio-to-sign language device 100 further includes a rendering module, which is further used to render a sign language animation and display the rendered sign language animation on a web page.

[0132] Each module in the above-mentioned audio-to-sign language device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0133] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an audio-to-sign language method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0134] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0136] Capture real-time audio on a web page; use a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; parse the text sequence to obtain multiple sign language movements; synthesize multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play the real-time audio.

[0137] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0138] The real-time audio is preprocessed, including noise suppression, automatic gain control, and echo cancellation; a pre-trained audio recognition model is used to perform speech recognition on the preprocessed audio to obtain a text sequence corresponding to the real-time audio.

[0139] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0140] Perform feature extraction on the preprocessed audio to obtain a feature vector that can be understood by the audio recognition model; use the pre-trained audio recognition model to perform speech recognition on the feature vector to obtain a text sequence corresponding to the real-time audio.

[0141] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0142] The text sequence is subjected to text preprocessing to obtain multiple processed text words, wherein the text preprocessing includes text cleaning, lexical analysis, grammatical analysis, and named entity recognition; each processed text word is converted into a corresponding semantic vector; the multiple semantic vectors are converted into action sequences through a pre-trained sequence-to-sequence model; and multiple sign language actions are determined based on the action sequences.

[0143] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0144] A preset sign language vocabulary library is obtained, in which the corresponding relationship between sign language vocabulary and sign language actions is stored; based on the preset sign language vocabulary library, multiple actions with a combination relationship in the action sequence are mapped to corresponding sign language actions to obtain multiple sign language actions.

[0145] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0146] Render the sign language animation and display the rendered sign language animation on the web page.

[0147] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0148] Capture real-time audio on a web page; use a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; parse the text sequence to obtain multiple sign language movements; synthesize multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play the real-time audio.

[0149] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0150] The real-time audio is preprocessed, including noise suppression, automatic gain control, and echo cancellation; a pre-trained audio recognition model is used to perform speech recognition on the preprocessed audio to obtain a text sequence corresponding to the real-time audio.

[0151] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0152] Perform feature extraction on the preprocessed audio to obtain a feature vector that can be understood by the audio recognition model; use the pre-trained audio recognition model to perform speech recognition on the feature vector to obtain a text sequence corresponding to the real-time audio.

[0153] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0154] The text sequence is subjected to text preprocessing to obtain multiple processed text words, wherein the text preprocessing includes text cleaning, lexical analysis, grammatical analysis, and named entity recognition; each processed text word is converted into a corresponding semantic vector; the multiple semantic vectors are converted into action sequences through a pre-trained sequence-to-sequence model; and multiple sign language actions are determined based on the action sequences.

[0155] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0156] A preset sign language vocabulary library is obtained, in which the corresponding relationship between sign language vocabulary and sign language actions is stored; based on the preset sign language vocabulary library, multiple actions with a combination relationship in the action sequence are mapped to corresponding sign language actions to obtain multiple sign language actions.

[0157] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0158] Render the sign language animation and display the rendered sign language animation on the web page.

[0159] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0160] Capture real-time audio on a web page; use a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; parse the text sequence to obtain multiple sign language movements; synthesize multiple sign language movements into a continuous sign language animation, display the sign language animation on the web page and play the real-time audio.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0162] The real-time audio is preprocessed, including noise suppression, automatic gain control, and echo cancellation; a pre-trained audio recognition model is used to perform speech recognition on the preprocessed audio to obtain a text sequence corresponding to the real-time audio.

[0163] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0164] Perform feature extraction on the preprocessed audio to obtain a feature vector that can be understood by the audio recognition model; use the pre-trained audio recognition model to perform speech recognition on the feature vector to obtain a text sequence corresponding to the real-time audio.

[0165] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0166] The text sequence is subjected to text preprocessing to obtain multiple processed text words, wherein the text preprocessing includes text cleaning, lexical analysis, grammatical analysis, and named entity recognition; each processed text word is converted into a corresponding semantic vector; the multiple semantic vectors are converted into action sequences through a pre-trained sequence-to-sequence model; and multiple sign language actions are determined based on the action sequences.

[0167] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0168] A preset sign language vocabulary library is obtained, in which the corresponding relationship between sign language vocabulary and sign language actions is stored; based on the preset sign language vocabulary library, multiple actions with a combination relationship in the action sequence are mapped to corresponding sign language actions to obtain multiple sign language actions.

[0169] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0170] Render the sign language animation and display the rendered sign language animation on the web page.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0172] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0173] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0174] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for converting audio to sign language, characterized in that: The method comprises: Capture live audio on web pages; Using a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio; Parsing the text sequence to obtain a plurality of sign language actions; The plurality of sign language actions are synthesized into a continuous sign language animation, the sign language animation is displayed on the webpage, and the real-time audio is played.

2. The method according to claim 1, characterized in that The using a pre-trained audio recognition model to perform speech recognition on the real-time audio to obtain a text sequence corresponding to the real-time audio includes: Preprocessing the real-time audio, wherein the preprocessing includes noise suppression, automatic gain control, and echo cancellation; The pre-trained audio recognition model is used to perform speech recognition on the pre-processed audio to obtain a text sequence corresponding to the real-time audio.

3. The method according to claim 2, characterized in that The method of using a pre-trained audio recognition model to perform speech recognition on the pre-processed audio to obtain a text sequence corresponding to the real-time audio includes: Perform feature extraction on the preprocessed audio to obtain a feature vector that can be understood by the audio recognition model; A pre-trained audio recognition model is used to perform speech recognition on the feature vector to obtain a text sequence corresponding to the real-time audio.

4. The method according to claim 1, characterized in that: The parsing of the text sequence to obtain a plurality of sign language actions includes: Performing text preprocessing on the text sequence to obtain a plurality of processed text words, wherein the text preprocessing includes text cleaning, lexical analysis, grammatical analysis, and named entity recognition; Convert each processed text word into a corresponding semantic vector; Convert multiple semantic vectors into action sequences through a pre-trained sequence-to-sequence model; Based on the action sequence, a plurality of sign language actions are determined.

5. The method according to claim 4, characterized in that Determining a plurality of sign language actions based on the action sequence includes: Acquire a preset sign language vocabulary library, wherein the preset sign language vocabulary library stores the correspondence between sign language vocabulary and sign language actions; Based on the preset sign language vocabulary library, multiple actions with a combination relationship in the action sequence are mapped to corresponding sign language actions to obtain multiple sign language actions.

6. The method according to claim 1, characterized in that The method further comprises: Rendering the sign language animation, and displaying the rendered sign language animation on the webpage.

7. An audio to sign language device, characterized in that: The device comprises: Capture module, used to capture real-time audio on web pages; A recognition module, used to perform speech recognition on the real-time audio using a pre-trained audio recognition model to obtain a text sequence corresponding to the real-time audio; A parsing module, used for parsing the text sequence to obtain a plurality of sign language actions; The display module is used to synthesize multiple sign language movements into a continuous sign language animation, display the sign language animation in the web page and play the real-time audio.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images