Voice text bidirectional conversion method and device, equipment and medium

Abstract

The invention relates to the technical field of voice processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a voice text bidirectional conversion method, device, equipment and medium, and the method comprises the steps: respectively executing voice recognition or voice synthesis operation according to the type of input information; for the voice information, noise suppression parameters are generated in combination with the lip movement video data, noise reduction processing is executed, and the recognition accuracy is improved; for text information, a pre-generated speaker style vector is obtained, the vector is cited in the speech synthesis process to generate natural personalized speech, and lip movement information and tactile feedback which are synchronous with speech output are generated. According to the method, complex noise is suppressed by fusing lip movement data, personalized voice is generated by using the style vector, and lip movement and touch information is output, so that bidirectional real-time conversion of voice and text in a complex environment is realized, and recognition accuracy, voice naturalness and interaction synchronism are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a method, device, equipment and storage medium for bidirectional conversion of speech to text. Background Art

[0002] In the development of barrier-free voice interaction technology, the performance of existing speech recognition (ASR) and speech synthesis (TTS) systems in dynamic noise environments still has significant deficiencies. Traditional noise suppression algorithms, such as spectral subtraction and Wiener filtering, mainly target steady-state background noise and have poor robustness when dealing with complex dynamic interference such as vehicle bumps and sudden horn honking. While reducing noise, these algorithms often destroy the high-frequency details of the voice signal, especially the key components of clarity such as sibilant sounds. This leads to a decrease in speech intelligibility and a subjective listening score (MOS) below 3.2. In this context, speech recognition accuracy is impaired, and the misrecognition rate can even reach over 40% under high-noise conditions, seriously affecting the interactive experience.

[0003] Existing systems often face the problem of excessively high interaction latency in two-way voice interaction. Although most commercial solutions have achieved full-duplex communication, the end-to-end recognition latency of the ASR model can exceed 500ms, and the audio synthesis of the TTS module also requires more than 200ms. This results in an overall round-trip speech-to-speech interaction latency exceeding 1 second, making it difficult to meet the international standard for real-time interaction requiring a response time of less than 200ms. In barrier-free application scenarios, such as providing real-time speech-to-lip movement conversion assistance for the hearing-impaired, this latency can lead to information synchronization deviations, making accurate lip reading and assisted perception impossible.

[0004] In terms of personalized expression and multilingual adaptation, existing speech synthesis systems are heavily dependent on data, usually requiring users to provide more than 30 hours of high-quality recording samples to complete voice cloning, making it difficult to generalize to ordinary users. Support for minority languages ​​or dialects (such as Cantonese and Dong languages) is insufficient, and the similarity evaluation score (SIM) of the synthesized speech is low, only 3.1, lacking personalized expression and language diversity. In cross-language recognition, the phoneme transfer ability is weak, and the confusion rate is increased by about 15% compared to the monolingual system, further limiting its practicality in multilingual environments.

[0005] In the fintech sector, voice interaction systems are often used in scenarios such as remote identity authentication, customer service, and business processing. However, high error rates and latency issues often lead to users experiencing repeated input or interrupted responses during account operations, transaction confirmations, or voice customer service, reducing customer satisfaction and service efficiency. Furthermore, the difficulty in adapting to diverse accents and contexts affects the system's universality and fairness across different regions and demographics.

[0006] In the healthcare sector, voice interaction is widely used for assisted communication, patient follow-up visits, and health education. It is particularly popular for hearing-impaired patients. However, limited by the high latency and multimodal input limitations of traditional voice systems, current technology struggles to achieve efficient and accurate information synchronization and personalized expression, hindering its adoption in real-world clinical scenarios. Summary of the Invention

[0007] The main purpose of the present invention is to provide a method, device, equipment and storage medium for bidirectional conversion of speech and text, aiming to solve the technical problem that the existing technology cannot achieve low-latency, high-fidelity and multimodal collaborative bidirectional real-time conversion of speech and text in a complex dynamic noise environment.

[0008] To achieve the above object, the present invention provides a method for bidirectional conversion of speech to text, comprising:

[0009] receiving input information to be converted and determining the type of the input information;

[0010] When the input information type is voice information, obtaining multimodal environment data including lip movement video data, and generating noise suppression parameters based on the multimodal environment data;

[0011] Performing noise reduction processing on the input information to be converted according to the noise suppression parameters to generate clear speech information;

[0012] Inputting the clear speech information into a speech recognition model and outputting initial text information;

[0013] generating lip movement compensation data based on the clear voice information and the lip movement video data, and correcting the initial text information using the lip movement compensation data to generate target text information;

[0014] When the input information type is text information, obtaining a pre-generated speaker style vector;

[0015] Inputting the input information to be converted into a text-to-speech model and referencing the speaker style vector, and outputting target speech information;

[0016] Lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information are generated.

[0017] Furthermore, to achieve the above-mentioned object, the present invention provides a device for bidirectional conversion of speech to text, comprising:

[0018] An input type identification module, configured to receive input information to be converted and determine the type of the input information;

[0019] a multimodal perception module, configured to, when the input information type is voice information, obtain multimodal environment data including lip movement video data, and generate noise suppression parameters based on the multimodal environment data;

[0020] A speech noise reduction module, configured to perform noise reduction processing on the input information to be converted according to noise suppression parameters to generate clear speech information;

[0021] A speech recognition module, configured to input the clear speech information into a speech recognition model and output initial text information;

[0022] a lip movement compensation module, configured to generate lip movement compensation data based on the clear voice information and the lip movement video data, and use the lip movement compensation data to correct the initial text information to generate target text information;

[0023] A style vector extraction module, configured to obtain a pre-generated speaker style vector when the input information type is text information;

[0024] a speech synthesis module, configured to input the input information to be converted into a text-to-speech model and reference the speaker style vector to output target speech information;

[0025] The feedback generation module is used to generate lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information.

[0026] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a speech-to-text bidirectional conversion program stored in the memory and runnable on the processor. When the speech-to-text bidirectional conversion program is executed by the processor, the steps of the speech-to-text bidirectional conversion method as described above are implemented.

[0027] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a speech-to-text bidirectional conversion program is stored. When the speech-to-text bidirectional conversion program is executed by a processor, the steps of the speech-to-text bidirectional conversion method described above are implemented.

[0028] Beneficial effects: The present invention relates to the field of speech processing technology and can be applied to business scenarios such as financial technology and medical health. A method, device, equipment and medium for bidirectional conversion of speech to text are disclosed, including: receiving input information to be converted and determining the type of input information; when the input information is speech information, obtaining multimodal environmental data including lip movement video data, generating noise suppression parameters based on the multimodal environmental data, and performing noise reduction processing to generate clear speech information; inputting the clear speech information into a speech recognition model and outputting initial text information; generating lip movement compensation data based on the clear speech information and lip movement video data, correcting the initial text information, and generating target text information; when the input information is text information, obtaining a pre-generated speaker style vector; inputting the input information to be converted into a text-to-speech model and referencing the speaker style vector, and outputting the target speech information; generating lip movement synchronization information synchronized with the target speech information and tactile feedback information corresponding to its energy characteristics. The present invention assists speech recognition by integrating lip movement video data, introduces speaker style vectors to improve the personalization of speech synthesis, and synchronously generates lip movement information and tactile feedback, effectively improving the recognition accuracy and naturalness of synthesized speech in complex noisy environments, reducing interaction delays, and enhancing the integrity and immersion of information transmission. It is suitable for multimodal barrier-free communication scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0030] Figure 1 A schematic diagram of an application environment of a method for bidirectional conversion of speech to text according to an embodiment of the present invention;

[0031] Figure 2 This is a flow chart of an embodiment of a method for bidirectional conversion of speech to text according to the present invention;

[0032] Figure 3 This is a functional module diagram of a preferred embodiment of the speech-to-text bidirectional conversion device of the present invention;

[0033] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0034] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] The method for bidirectional conversion of speech to text provided by the embodiment of the present invention can be applied in the following situations: Figure 1In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can receive input information to be converted through the user terminal and determine the type of input information; when the input information is voice information, obtain multimodal environment data including lip movement video data, generate noise suppression parameters based on the multimodal environment data, and perform noise reduction processing to generate clear voice information; input the clear voice information into the voice recognition model and output initial text information; generate lip movement compensation data based on the clear voice information and lip movement video data, correct the initial text information, and generate target text information; when the input information is text information, obtain a pre-generated speaker style vector; input the input information to be converted into text into the voice model and reference the speaker style vector to output the target voice information; generate lip movement synchronization information synchronized with the target voice information and tactile feedback information corresponding to its energy characteristics. The present invention assists voice recognition by integrating lip movement video data, introduces speaker style vectors to improve the personalization of voice synthesis, and synchronously generates lip movement information and tactile feedback, effectively improving the recognition accuracy and naturalness of synthesized voice in complex noise environments, reducing interaction delays, and enhancing the integrity and immersion of information transmission. It is suitable for multimodal barrier-free communication scenarios. The user end may be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server end may be implemented as an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below using specific embodiments.

[0037] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the method for bidirectional conversion of speech to text provided by the present invention. It should be noted that although the flow chart shows a logical order, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0038] like Figure 2 As shown, the speech-to-text bidirectional conversion method proposed by the present invention includes the following steps:

[0039] S10, receiving input information to be converted and determining the type of the input information;

[0040] In this embodiment, the process of receiving input information to be converted and determining the type of input information is intended to build a dynamic recognition mechanism for distinguishing the modal attributes of the input content and providing structured input labels for subsequent processing flows. Input information refers to the original data stream submitted by the user through the terminal, which may include audio signals, text instructions or a combination thereof. The information is collected through an input interface module, which can be connected to various types of input devices, including microphones, touch screens, keyboards or remote input channels. In order to ensure the integrity and consistency of the input content, a data encoding detection mechanism is usually present when receiving input information to determine whether the input format meets the system processing requirements, such as whether it is a valid audio stream or a character sequence that complies with language encoding specifications.

[0041] The determination of the input information type depends on the input recognition unit, which is usually composed of a data parsing module and a modal discrimination module. The data parsing module extracts basic dimensions based on the data structure features of the input information, such as packet headers, timing fluctuations, or character encoding, and preliminarily classifies its structure through rule judgment or a lightweight neural network model. The modal discrimination module further analyzes the extracted features to identify whether the input is audio, text, or other data modalities. For audio types, statistical features such as short-time energy, zero-crossing rate, and voice activity detection can be used to confirm whether it contains recognizable human speech patterns. For text types, it is necessary to parse the character set and verify the language model confidence or semantic structure integrity.

[0042] In addition, an input context recognition mechanism can be introduced in the modal discrimination process to model the current input environment or historical input records, thereby improving judgment accuracy. For example, when the system is in a hearing-impaired assistance scenario, the historical interaction frequency and input device type may affect the selection of the current modal judgment strategy. In order to enhance real-time performance and stability, the modal judgment module should be compressed and optimized when embedded in the device deployment, using a pruned model, a low-bit quantization model, or a lightweight variant of the Transformer structure to control the model response time within a preset range.

[0043] In the actual deployment process, unified reception of input information can be achieved by integrating an input hub component in the terminal device that operates together with an audio acquisition module and a text input channel. For audio information, the microphone array is used to collect and perform end-side pre-enhancement processing, and then transmitted to the central processing unit to determine its modal properties. For text input, the character stream can be connected to the text recognition submodule through manual entry, copy and paste, or remote server push. A lightweight neural network structure, such as MobileNet or TinyBERT, can be used inside the modal recognition module to identify the modal features of the input signal and output a signal label used to control the execution path of subsequent modules. In high-computing resource scenarios, a semantic recognition module can be further superimposed to determine the dominant modality of the multimodal mixed input and implement a main modality priority processing strategy.

[0044] In actual operation, for specific scenarios, such as mixed-modality inputs involving both audio and text, the system can set priority decision logic, for example, prioritizing voice inputs containing lip movement video data to ensure accurate execution of downstream functions such as noise reduction and semantic recognition. This decision logic is configured through a rules engine, combining input modality confidence and user preference tags to dynamically adjust the modality routing strategy.

[0045] Example: In healthcare, this process can be used to assist hearing-impaired users with real-time communication. The system captures voice input through an external microphone and triggers lip movement analysis and speech recognition based on input modality, enabling patients to receive real-time text feedback on their medical advice. Before speech is converted to text, a type determination mechanism ensures that the captured content is human speech and not ambient noise, improving subsequent recognition accuracy.

[0046] In the financial sector, such as in intelligent teller machine (ATM) scenarios, when users make voice requests for operations such as transfers and inquiries, the system identifies these as voice input through a modal identification module and assigns them to a voice recognition path. This identification process improves the financial service system's tolerance for non-standard input, reduces the risk of process interruptions caused by input misjudgment, and lowers the operational learning curve for customers.

[0047] This embodiment, by constructing a receiving mechanism with modal recognition and type determination functions at the input stage, can provide clear, high-confidence modal labels for subsequent processes, significantly reducing the modal recognition error rate. At the same time, it effectively avoids processing errors such as misclassification of speech as text or vice versa, and enhances the robustness of input information processing. The lightweight design of the modal judgment structure ensures operational efficiency in low-power devices, allowing the entire system to be deployed in resource-sensitive scenarios such as hearing-impaired assistance terminals to meet low-latency real-time communication requirements.

[0048] S20, when the input information type is voice information, obtaining multimodal environment data including lip movement video data, and generating noise suppression parameters based on the multimodal environment data;

[0049] In this embodiment, after the input information is recognized as voice information, the system will actively call on multi-channel perception resources related to voice processing to construct an input data set that integrates multiple modalities. Lip movement video data refers to a dynamic video frame sequence that captures the movement details of the user's lip area, acquired through an image acquisition device such as an RGB camera, an infrared camera module, or a depth camera. This sequence is synchronized with the voice signal on the timeline and is used to capture pronunciation-aiding information during the speech production process. Lip movement video data contains structural features such as lip opening and closing, up and down movement, and mouth shape changes, and has a reference value for compensating for situations where the voice is unclear or there is severe background noise interference.

[0050] Multimodal environmental data refers to a composite input data structure that combines voice capture data and lip movement video data. It typically also includes other available modal data, such as inertial sensor signals (for detecting head vibration and mouth movement), depth information, or optical flow maps. This data is integrated into a tensor structure suitable for model processing through operations such as time alignment, data format unification, and dimensionality mapping, and serves as input for subsequent multimodal neural networks.

[0051] The noise suppression parameters are generated by inputting the aforementioned multimodal environmental data into a trained multimodal neural network model, which consists of multiple branches: a main branch processes the acoustic signal and generates a basic mask, and an auxiliary branch processes the lip movement video data and generates a corresponding pronunciation feature map. During the training phase, the model learns the synergy between different modalities, thereby eliminating non-speech components and enhancing speech areas. The final output noise suppression parameters may include a masking weight matrix in the time-frequency domain, a speech probability map, or specific frequency band adjustment coefficients, which are subsequently used to control the noise reduction process.

[0052] In its implementation, upon recognizing voice modal input, the system immediately activates the video capture channel and specifies a capture area to lock onto the user's face. The image acquisition device captures the lip movement video data stream at a frame rate of 30 frames per second, while simultaneously caching the synchronized audio signal at a sampling rate of 16kHz. The system then aligns the timestamps of the data in the two data channels to ensure a one-to-one match between each video frame and the corresponding voice frame. To improve fusion efficiency, a frame-level fusion method can be used to construct a multimodal tensor within the time window, which serves as the unified input for the neural network.

[0053] The neural network model used to process this data adopts a two-branch structure: a convolutional neural network for generating acoustic masks, and a branch for convolutional encoding and graph modeling of lip trajectories. The model training process integrates CTC loss, perceptual loss, and adversarial loss to ensure noise robustness and speech integrity. During deployment, the model is compressed and pruned to enable it to run on edge devices or mobile terminals, meeting the low resource consumption requirements of barrier-free terminals.

[0054] An example: In healthcare, when a doctor speaks to a hearing-impaired patient in a noisy exam room, traditional speech recognition systems can be disrupted by background noise, such as the sound of an ECG monitor, causing recognition failure. The system captures the doctor's lip movements through a camera and feeds this information into a multimodal model along with the speech signal. This generates high-confidence noise suppression parameters, improving speech clarity to a level suitable for real-time text feedback and making it easier for the patient to understand.

[0055] In financial scenarios, such as when a user issues a voice query at an ATM or smart teller machine, interference from traffic and movement can cause speech recognition errors. By capturing images of the user's lip movements and integrating them with the audio signal, the system effectively eliminates background interference, significantly improving speech recognition accuracy and ensuring the accuracy of financial operations and smooth interactions.

[0056] This embodiment, by integrating lip movement video data, can effectively compensate for the insufficient recognition accuracy of pure audio signals in strong interference environments. The lip movement images provide spatiotemporal information at the phoneme level, compensating for the limitations of traditional spectral subtraction or Wiener filtering in suppressing speech details. The noise suppression parameters generated by multimodal environmental data have stronger dynamic adaptability and can dynamically adjust the spectral mask in the presence of sudden noise (such as car horns and crowd noise), thereby achieving high-precision noise reduction without compromising speech clarity, effectively improving the input quality of the speech recognition module.

[0057] S30, performing noise reduction processing on the input information to be converted according to the noise suppression parameter to generate clear speech information;

[0058] In this embodiment, after the multimodal neural network generates noise suppression parameters, the system applies these parameters to the input voice data to achieve effective noise reduction. The input information to be converted refers to the original voice data provided by the user in a multimodal environment, which may contain various background noise components such as wind noise, electrical noise, traffic noise, and human voice interference. To improve the signal-to-noise ratio, this voice data needs to be divided into multiple short time frames for processing.

[0059] Each frame of speech data is converted to a frequency domain representation using a short-time Fourier transform (STFT), forming a speech spectral signature consisting of dimensions such as the amplitude and phase spectra. The system applies a noise reduction mask, defined in the noise suppression parameters, to the amplitude dimension of the spectrum to shield or attenuate background noise. This mask is typically a matrix consisting of floating-point values ​​between [0 and 1], where smaller values ​​indicate a higher likelihood of noise at that frequency point.

[0060] In addition to the basic mask, spectral offset compensation parameters can be used to adjust for frequency drift in the speech spectrum caused by environmental changes, such as head movement or changes in microphone reception angle. The system further compensates and corrects the processed spectral features and reconstructs them back into the time domain. This reconstruction operation preserves the phase information of the original speech and, combined with the adjusted amplitude spectrum, reconstructs the time series speech signal through an inverse Fourier transform.

[0061] Ultimately, all noise-reduced speech frames are spliced ​​together in timeline order to form a continuous segment of clear speech information, which is used for subsequent speech recognition, style modeling, or other acoustic processing tasks. This clear speech retains the linguistic components and prosodic structure of the original speech while significantly reducing non-speech interference.

[0062] In one implementation, the system first segments each segment of input speech data into 20-millisecond frames with a 10-millisecond frame shift, and then uses a Hamming window for weighting. Each frame undergoes an FFT transform to generate a 512-dimensional spectral feature matrix, which serves as the basis for processing. A noise reduction mask generated by the neural network is point-by-point multiplied by the spectral features. Linear interpolation is then applied to each frame using a spectral shift parameter to correct for mismatched frequency bands caused by the shift.

[0063] During the spectrum recovery phase, an inverse STFT method, symmetric to the STFT, is used to perform time-domain reconstruction. To reduce edge distortion, the system splices frame data using an overlap-addition method and applies a weighted smoothing function to the transitions in the seam areas. To accommodate mobile device resource constraints, the system employs a sparse mask encoding strategy during the noise reduction phase, applying fine masking only to high-energy frequency bands to reduce redundant computations.

[0064] Different noise reduction model accuracies can also be selected based on the computing power conditions of the target platform. For example, a low-power terminal can load a lightweight network, retaining only the mask details of the first three frequency bands, sacrificing some voice fidelity in exchange for fast response capabilities.

[0065] Example: In healthcare scenarios, doctors and medical staff can accurately complete medical records or convey clinical instructions through voice input, even in the presence of complex background noise such as ventilator alarms, suction equipment, and conversations. The system dynamically removes the spectral information of non-speech sound sources using noise suppression parameters, and performs fidelity-prioritized noise reduction and reconstruction on the frequency characteristics of the doctor's voice signal, generating clear voice information in real time. This prevents the inadvertent elimination of high-frequency consonants and oral plosives, providing a stable input foundation for subsequent speech recognition and lip movement compensation.

[0066] In financial business scenarios, users make voice requests, such as account balance inquiries and business processing instructions, at business halls, ATMs, or smart customer service terminals. These requests are often made in crowded, noisy environments, or surrounded by background broadcasts. By applying noise suppression parameters to the voice signal's frequency domain, the system dynamically identifies the semantically dominant frequency band and suppresses non-target sound sources, achieving rapid noise reduction of the customer's voice signal. This process not only improves recognition accuracy but also significantly reduces semantic misunderstandings and false feedback caused by background noise, thereby ensuring a smooth voice interaction experience and the security of financial operations.

[0067] This embodiment significantly improves the pertinence and flexibility of speech noise reduction by applying multimodally generated noise suppression parameters to frequency domain processing. Compared to traditional static masking methods, this processing method can dynamically respond to different types of noise scenarios, preserving the semantic integrity of speech signals while effectively reducing nonlinear noise. Recognition performance in complex environments is significantly improved, while avoiding speech distortion caused by excessive noise reduction, effectively maintaining subsequent speech recognition accuracy and the naturalness of the user interaction experience.

[0068] S40, inputting the clear voice information into a voice recognition model and outputting initial text information;

[0069] In this embodiment, clear speech information refers to the speech data stream formed after noise interference suppression and spectrum restoration processing. It has good fidelity in terms of temporal continuity, spectral stability, and speech integrity, and can meet the requirements of subsequent semantic decoding. The speech recognition model is an automatic speech recognition architecture built on a deep neural network. It typically includes a feature extraction module, an acoustic modeling module, and a language model integration module. Its purpose is to convert continuous speech signals into structured character sequences or word sequences to form initial text information.

[0070] To input clear speech information into the speech recognition model, the input data must first be framed and subjected to short-time Fourier transform processing to extract time-frequency domain features such as Mel-frequency cepstral coefficients (MFCCs), spectrograms, and cepstrograms, forming a feature vector sequence that can be analyzed by deep learning models. Then, through an end-to-end neural architecture, such as a convolutional recurrent network based on the CTC (Connectionist Temporal Classification) loss function or a Transformer model using an attention mechanism, the time series features are mapped to a corresponding phoneme label sequence or word label sequence, and combined with language model constraint correction, the initial text information is output. This initial text information is the recognition result without contextual reordering or external correction.

[0071] Initial text information serves as the fundamental input for subsequent lip movement compensation and semantic restoration. Its semantic integrity and phoneme accuracy significantly impact the stability of subsequent processing modules. Therefore, ensuring that the speech recognition model's input features are compatible with the frequency domain structure of clear speech information can significantly improve recognition accuracy and the robustness of subsequent tasks.

[0072] In specific implementations, the speech recognition model can adopt a CTC model structure based on a bidirectional long short-term memory network (BiLSTM) to support global modeling of temporal context, or a Conformer architecture based on a self-attention mechanism to improve the stability of long sentence recognition by simultaneously modeling local speech details and global semantic relationships. Audio input can be collected through a microphone array and then dynamically normalized and frequency-expanded to ensure that the energy distribution of clear speech information is balanced in the feature space, adapting to the structural requirements of the model input tensor.

[0073] For multi-party speech scenarios, the system uses a channel selection mechanism based on a sound source separation algorithm to prioritize the target speaker's clear speech signal and feed it into the recognition model. For mobile deployments, a lightweight speech recognition engine (such as Distill-ASR or Wav2Vec-lite) can be used to reduce model complexity and runtime latency. Furthermore, a sub-word output mode with timestamps can be configured to provide precise time alignment information for the subsequent lip movement compensation module.

[0074] Example: In healthcare, when a hearing-impaired user enters an outpatient clinic, while the doctor explains the diagnosis or medication plan, the wearable device they are wearing inputs clear, pre-noise-reduced speech information into a speech recognition model. This model then outputs the initial text information in real time and displays it simultaneously on the user's terminal, allowing the hearing-impaired user to understand the doctor's original statement without relying on manual sign language interpretation. This approach significantly shortens the information delivery path between doctors and patients, reducing communication costs caused by repeated explanations or misunderstandings, and is particularly valuable in emergency and high-pressure situations.

[0075] In financial services scenarios, when hearing-impaired customers visit the teller to purchase financial products or conduct risk assessments, the system feeds the teller's spoken instructions into the speech recognition model in real time. The resulting initial text is then projected directly into the user's visual interface, helping them understand key business terms and decision-making information and avoiding financial misunderstandings or disputes caused by semantic discrepancies. This step ensures the integrity and timeliness of the recognized text, especially in the noisy environment of bank branches, supporting the accessibility and fairness of financial services.

[0076] By feeding clear speech information into a deep speech recognition model, this embodiment significantly improves the accuracy and robustness of speech-to-text conversion, effectively avoiding recognition biases caused by environmental interference, speaker characteristics, or speech rate variations. This provides a high-quality text foundation for subsequent data correction based on lip movement compensation. This also improves the semantic starting point quality of the subsequent text generation module and reduces redundant repair costs.

[0077] S50, generating lip movement compensation data based on the clear voice information and the lip movement video data, and using the lip movement compensation data to correct the initial text information to generate target text information;

[0078] In this embodiment, clear speech information and lip movement video data are used as input data sources and jointly participate in the processing process. The two types of data have high temporal synchronization on the time axis, ensuring consistency between the speech content and the pronunciation mouth shape. Clear speech information is extracted through spectral modeling to extract time-frequency spectrum features. Typically, methods such as short-time Fourier transform (STFT) can be used to obtain the energy spectrum and intonation change trends corresponding to each frame. Lip movement video data is analyzed by the video processing module to analyze the facial area, and further extracts the three-dimensional mouth shape key points corresponding to the mouth area in each frame based on the key point detection model, such as the upper lip, lower lip, mouth corner position, and degree of opening and closing.

[0079] The time-frequency spectrum features of clear speech information and the sequence of lip shape key points from the lip movement video data are aligned along a unified time axis. Frame-level correspondence can be achieved using methods based on dynamic time warping (DTW) or feature correlation mapping. The alignment results are used to generate lip movement compensation data. This compensation data is constructed by modeling the complementarity between audio and visual signals during pronunciation. This data is used to correct similar phonemes that cannot be accurately distinguished due to the limitations of the audio recognition model.

[0080] Initial text information, as an intermediate product of speech recognition, may contain some phonemes that may be confused due to high-frequency loss or ambiguity in resonance. Combined with the generated lip movement compensation data, the recognized text can be reviewed syllably, particularly replacing or confirming easily confused plosives and fricatives. Ultimately, target text information is generated that better reflects the actual pronunciation.

[0081] In actual deployment, a lightweight three-dimensional facial key point detection model (such as MediaPipe FaceMesh) can be used to extract the lip shape of each frame and construct a time series of mouth shape changes. This time series is time-aligned at the frame level with the speech spectrum obtained by the Mel spectrum extractor to calibrate the correspondence between the speech frame and the mouth shape frame. When generating lip movement compensation data, the trained multimodal alignment model can be used to achieve collaborative modeling of audio and vision. For example, a gated attention mechanism is used to strengthen the coupling relationship between phonemes and mouth shapes. For syllables with high confusion in the initial text information (such as zh-ch-sh, dt, etc.), cross-validation is performed by identifying the mouth shape information of the corresponding time period. If it does not match, it is replaced with the corresponding character of the phoneme that is more consistent with the mouth shape characteristics.

[0082] The implementation can also include a custom obfuscated phoneme dictionary, built based on acoustic similarity and lip shape differences, to accelerate post-recognition correction. After phoneme-level correction, the target text is verified for semantic consistency using a language model to ensure contextual fluency.

[0083] Example description: In a medical and health scenario, hearing-impaired users receive verbal information from doctors through head-mounted cameras and audio acquisition units. Since hospital environments are often accompanied by interference such as background broadcasts and equipment sounds, audio data is difficult to be completely reliable. The system collects the user's frontal lip movement image and synchronously collects audio data, and then inputs it into the recognition engine for joint modeling. When the doctor issues key instructions such as "take medicine" and "taboos", even if some syllables in the audio are lost due to background noise, the lip movement compensation mechanism can still recognize the true meaning based on the changes in the mouth shape and present it to the user terminal, ensuring accurate medical communication.

[0084] In financial business scenarios, when hearing-impaired users communicate with bank tellers or online customer service regarding product details, the system automatically captures speech and lip movement signals. Because financial terminology is often similar and easily confused, phonetically similar words like "return," "default," and "risk" are more likely to be misidentified. By comparing lip movement features, the system accurately identifies the actual pronunciation and generates corrected target text information for the customer's reference. This capability significantly reduces the risk of misunderstandings in transaction interpretations and errors in contract signing, improving the accessibility and trust of financial services for hearing-impaired users.

[0085] This embodiment effectively improves the robustness of audio recognition in high-noise environments by introducing lip movement compensation data into the text recognition process. Even if there are problems such as resonance ambiguity and masking interference in the audio, using lip shape information for speech recognition compensation can significantly reduce phoneme misrecognition and improve the accuracy and readability of the overall text output. This mechanism overcomes the disadvantages of traditional ASR in a purely acoustic path, enabling text recognition to have cross-modal information perception capabilities, making it particularly suitable for barrier-free communication scenarios.

[0086] S60, when the input information type is text information, obtaining a pre-generated speaker style vector;

[0087] In this embodiment, when the input information type is determined to be text, the system invokes a pre-generated speaker style vector to achieve personalized speech synthesis that aligns with the user's intent. A speaker style vector is a numerical representation of expressive features such as timbre, intonation, rhythm, and emotion used in speech synthesis. It is typically composed of a multi-dimensional dense vector, with each dimension encoding a different speech performance attribute. To ensure that the synthesized speech matches the user's identity or specific interaction scenario, a mapping relationship corresponding to the user's identity must first be established.

[0088] During implementation, the system uses the current user's identity (such as user account, device ID, session context identifier, etc.) as a search index to query and extract the corresponding speaker style vector from a pre-built speaker style vector database. This database can be pre-set with multiple style vectors, such as gentle, friendly, professional, authoritative, and childlike, to cover different user preferences and application needs.

[0089] To avoid calling invalid or expired vector representations, the obtained speaker style vector must be verified for validity after extraction. Verification methods may include integrity checks (checking for missing dimensions), timeliness checks (confirming whether the vector is the current version), or automatic evaluation of audio synthesis samples (such as determining scoring thresholds), to ensure the stability and consistency of the style vector used in subsequent speech synthesis processes. After verification, the speaker style vector is loaded and prepared for injection into the text-to-speech model, serving as a modulation parameter for subsequent speech synthesis steps.

[0090] One implementation method is to extract a style vector based on historical speech data provided by a specific speaker through a trained speaker encoding model (such as a d-vector or x-vector architecture), and then bind the vector to the speaker ID and store it in a database. Whenever the input information type is text, the system first identifies the current user identity or contextual session state, and then calls the corresponding style vector. The style vector can also be classified according to context, such as "formal tone", "relaxed tone", "children's style", etc., with multiple preset types for selection.

[0091] Another implementation method is a device-side caching mechanism. With user authorization, the style vector from the user's previous voice interaction is stored locally on the device. This vector can be referenced later even when no voice sample is available, ensuring consistent output style. For the first user interaction, a style template can be selected from a public dataset for initial migration.

[0092] The style vector can also be adaptively selected based on the type of synthesis scenario, such as using a formal and steady tone in financial consultations, and a friendly and moderately spoken tone in medical consultations, to achieve adaptation of content and style.

[0093] Example: In the healthcare sector, hearing-impaired individuals enter text related to diagnosis and treatment into hospital reception terminals or assistive devices, such as describing physical discomfort, asking doctors how to take medication, or making appointments. After the system identifies the input as text, it retrieves pre-generated speaker style vectors, such as those describing friendly and gentle voice styles, and converts the input text into speech output in real time. This allows doctors to understand the hearing-impaired user's needs through natural and fluent speech, significantly improving the efficiency of two-way communication.

[0094] In the financial services sector, hearing-impaired users use keyboards to input text questions about loan rates, wealth management products, or business inquiries into branch terminals or mobile banking. The system uses pre-generated speech vectors with a formal and steady tone to synthesize these textual contents into audio, enabling tellers or customer service staff to accurately understand the user's needs through aural listening. This improves communication efficiency and ensures that key information is not missed during interactions, meeting the high accuracy and service consistency requirements of financial services.

[0095] This embodiment obtains and applies pre-generated speaker style vectors in the text-to-speech process, allowing the system to improve the personalization and naturalness of speech output while maintaining high generation efficiency. This structure makes the synthesized speech no longer single and mechanical, but presents a speaking style with recognition and emotional color. It is particularly suitable for scenarios where auditory trust needs to be established, such as barrier-free assistance, voice customer service, digital assistants, etc. This avoids the repeated calculation of re-extracting the style for each input, improving overall operating efficiency and user experience consistency.

[0096] S70, inputting the input information to be converted into a text-to-speech model and referencing the speaker style vector to output target speech information;

[0097] In this embodiment, after text input undergoes type determination and style vector extraction, it enters the speech synthesis process as the object to be processed. This processing relies on a text-to-speech model, whose primary function is to map character-level or phoneme-level language representations into audible speech signals. A text-to-speech model typically includes multiple submodules, such as a text pre-processing module, a prosody modeling module, an acoustic model module, and a neural vocoder module, each of which plays a key role in converting input text into a speech waveform.

[0098] To achieve output that matches a specific expression style, a style control mechanism must be introduced into the speech synthesis process. The speaker's style vector is injected into the style modulation path within the model structure, typically before the acoustic feature modeling stage, to adjust the intonation, rhythm, and timbre parameters of the phoneme sequence. For example, after parsing the text into a phoneme sequence, the style vector is fed as an additional condition into an embedding layer or conditional normalization network to modulate the feature generation process, ensuring that the output matches the target expression style.

[0099] The acoustic model is responsible for outputting continuous acoustic features (such as Mel-spectrograms, F0, and energy features) based on the modulated phoneme sequence. These features carry the subjective perceptual attributes of speech. Subsequently, the neural vocoder receives the acoustic features and performs a waveform restoration process from the features, generating the final audio signal with high fidelity. This vocoder typically uses a neural network architecture, such as WaveNet and HiFi-GAN, which integrates time-dependent modeling and perceptual discrimination capabilities into its structure, making the generated speech superior to traditional methods in terms of naturalness and clarity.

[0100] By building an end-to-end multi-module collaborative system, text input can be converted and annotated as the main input, and the style vector can be connected to the model's style control layer as a conditional input. The style control layer can use conditional batch normalization, gated attention mechanism, or conditional variational autoencoder network structure to make the style vector influence the model's latent space expression. To optimize the modulation effect, weight parameters can be trained separately for different style vectors, or a multi-style adversarial network structure can be introduced to distinguish the expression capabilities of different styles. After the acoustic features are output, a highly efficient neural vocoder is used to reconstruct the target speech. The speech output module can be deployed on the terminal or cloud side for real-time or near-real-time synthesis.

[0101] For different language environments, the phoneme set and intonation encoding module can be expanded to accommodate multilingual text input, and the language label dimension can be added to the style vector to achieve adaptive style transfer in cross-lingual scenarios. If the target terminal has limited computing resources, distillation models or parameter pruning can be used to reduce model size, improve synthesis speed, and reduce energy consumption.

[0102] Example description: In a healthcare scenario, the text request input by a hearing-impaired user is recognized by the system and a soothing style vector is invoked. The generated voice has a smooth and gentle tone, enabling accompanying doctors or caregivers to communicate and respond in a more empathetic manner. This is particularly suitable for inquiries about chronic diseases, child care, or psychological rehabilitation scenarios.

[0103] In the financial field, hearing-impaired customers use the interface to input account opening requests. Based on their set formal style preferences, the system generates professional and clear voice output content by introducing a style vector with formal intonation, enabling teller staff or self-service terminals to accurately understand the requests and maintain service standardization, avoiding misunderstandings or delayed processing due to ambiguous voice emotions.

[0104] This embodiment implements personalized speech synthesis by integrating text input and a speaker's style vector into a text-to-speech model. The output speech possesses specified intonation and timbre characteristics, enhancing the listening experience and user recognition. Furthermore, the introduction of style vectors allows speech synthesis to better meet the emotional and attitude requirements of diverse interaction scenarios, improving the expressiveness and adaptability of synthesized speech and effectively supporting personalized services and multi-scenario interactions.

[0105] S80: Generate lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information.

[0106] In this embodiment, once the target speech information is generated, to enhance the visual and perceptual experience of voice interaction, it is necessary to extract synchronously driven auxiliary output information from it. This includes lip synchronization information for video expression and tactile feedback information for multimodal feedback. Lip synchronization information refers to lip shape parameters or lip motion curves derived based on the alignment between speech content and timing. This information is used to drive the lip animation of virtual avatars, 3D facial models, or digital characters, enabling the recipient to perceive the speech content through visual means. This is particularly suitable for users who are unable to perceive speech.

[0107] This process typically includes phoneme-lip shape mapping modeling, timing alignment algorithms, and lip movement parameter generation mechanisms. The speech signal is first mapped into a phoneme sequence, and static lip shape data is generated based on a pre-established phoneme-to-lip shape mapping table or training model. Subsequently, interpolation, smoothing, or a motion generation network is used to generate a continuous lip movement sequence, combining the speech's temporal boundary information and articulation rhythm. Lip movement synchronization information can be stored as 3D mesh vertex parameters, 2D keypoint trajectories, or BlendShape coefficients for downstream visualization modules.

[0108] At the same time, in order to enhance the tactile perception dimension of speech output, the energy features in the target speech information can be further extracted. The energy features are generally obtained by performing a short-time Fourier transform on the audio signal and calculating the sum of the squares of the amplitudes of each frame to obtain an energy envelope curve. This curve can reflect the fluctuations in the intensity of the speech and is suitable for modulating the tactile feedback intensity output by the vibrator. By normalizing the energy features, compressing the dynamic range and performing rhythm filtering, tactile modules such as vibration motors and force feedback devices can be driven to convey the strength and rhythm changes of the speech to the user, thereby supplementing the perception path under hearing loss.

[0109] A speech-driven animation synthesis network based on time alignment extracts phoneme boundaries and accent rhythm from the target speech. This is then combined with a pre-trained phoneme-lip shape model to generate lip parameters for each frame. This model, trained using convolutional neural networks or graph neural networks, supports multiple languages ​​and personalized lip shapes. The output lip movement information can be passed via a parameter stream to the avatar driver engine or 3D model rendering module for real-time lip synchronization. Energy features are acquired by processing the audio signal with a window function of 25ms frame length and 10ms frame shift, calculating the energy of each frame and constructing a continuous envelope. A dynamically adjustable threshold can be set to map the energy value to a haptic feedback intensity level. A rhythm filter is then used to smooth tactile stimulus variations and avoid short-term vibration interference. The generated haptic feedback signal can be encapsulated into a standard HAPT data structure and transmitted via Bluetooth or a wired channel to a wearable or embedded haptic device for localized stimulation of the user's skin surface.

[0110] In specific deployment, lip movement information can be rendered by configuring a visualization front-end that supports WebGL, and the tactile feedback hardware module can be controlled using a low-power MCU to ensure that the system can be deployed lightweight in mobile terminals or auxiliary devices to meet design requirements such as wearable and portable devices.

[0111] Example: In a barrier-free communication scenario, a hearing-impaired user receives a voice response via a mobile device. The system automatically generates corresponding lip movement animations based on the voice, presenting them on the user's screen as a virtual character lip-syncing the semantic content, enhancing their intuitive understanding of the language. Simultaneously, the system modulates the vibration rhythm of the handheld device based on the rhythm and energy intensity of the voice, allowing the user to sense the rhythmic center of gravity through their hands during changes in speech speed and emphasis, thus compensating for the lack of perception in voice reception.

[0112] In financial service scenarios, after a hearing-impaired customer initiates a voice inquiry through text, the system's synthesized voice feedback is accompanied by lip-synced animation, which is used by counter service staff to provide understanding and support at the silent communication window; if the hearing-impaired user wears a smart wristband or vibration prompt device, the system maps the audio energy into rhythmic vibration pulses, allowing the user to perceive the emphasis or transition content of the speech, thereby improving interaction efficiency and service experience.

[0113] This embodiment simultaneously generates lip-sync information and tactile feedback, providing the target speech with both visual and tactile enhancements. This significantly improves user experience and interactive responsiveness in scenarios where audio is unavailable or limited. Lip-sync information makes visual alignment and semantic reasoning more intuitive, while tactile feedback provides an alternative path for rhythm and accent perception, enabling equivalent reconstruction of speech content in silent perception.

[0114] The present invention relates to the field of speech processing technology and can be applied to business scenarios such as financial technology and medical health. A method, device, equipment and medium for bidirectional conversion of speech to text are disclosed, including: receiving input information to be converted and determining the type of input information; when the input information is speech information, obtaining multimodal environmental data including lip movement video data, generating noise suppression parameters based on the multimodal environmental data, and performing noise reduction processing to generate clear speech information; inputting the clear speech information into a speech recognition model and outputting initial text information; generating lip movement compensation data based on the clear speech information and lip movement video data, correcting the initial text information, and generating target text information; when the input information is text information, obtaining a pre-generated speaker style vector; inputting the input information to be converted into a text-to-speech model and referencing the speaker style vector, and outputting target speech information; generating lip movement synchronization information synchronized with the target speech information and tactile feedback information corresponding to its energy characteristics. The present invention assists speech recognition by integrating lip movement video data, introduces speaker style vectors to improve the personalization of speech synthesis, and synchronously generates lip movement information and tactile feedback, effectively improving the recognition accuracy and naturalness of synthesized speech in complex noisy environments, reducing interaction delays, and enhancing the integrity and immersion of information transmission. It is suitable for multimodal barrier-free communication scenarios.

[0115] In one embodiment, the above step S20 includes:

[0116] S201, when the input information type is voice information, start the audio acquisition channel, the video acquisition channel and the inertial sensor acquisition channel;

[0117] S202, acquiring an audio data stream through an audio acquisition channel, acquiring a lip movement video data stream through a video acquisition channel, and acquiring an inertial sensor data stream through an inertial sensor acquisition channel;

[0118] S203, integrating the audio data stream, the lip movement video data stream, and the inertial sensor data stream to generate multimodal environmental data;

[0119] S204, inputting the multimodal environment data into a multimodal neural network;

[0120] S205, processing the audio data stream in the multimodal environmental data through the main branch of the multimodal neural network to generate a basic noise reduction mask;

[0121] S206, processing the inertial sensor data stream in the multimodal environmental data through the auxiliary branch of the multimodal neural network to generate a spectrum shift compensation parameter;

[0122] S207: Fusing the basic noise reduction mask and the spectrum shift compensation parameter to generate a noise suppression parameter.

[0123] In this embodiment, upon determining that the input information type is voice information, a synchronous acquisition mechanism encompassing multiple modalities must be immediately activated to ensure that data from different modalities remain aligned on the timeline, laying the foundation for subsequent multimodal information processing. The audio acquisition channel is used to capture raw voice signal data in real time, typically through an electret microphone array, MEMS microphone, or array microphone array. The video acquisition channel uses a high-frame-rate camera to capture a sequence of lip movement images of the speaker, with a sampling frequency generally not less than 30 fps, to effectively distinguish lip shape variation characteristics between phonemes. The inertial sensor acquisition channel uses an IMU device (including an accelerometer and gyroscope) to sense the motion state of the acquisition device and environmental disturbances, capturing non-steady-state changes such as jitter, flipping, and impact.

[0124] Through a unified timestamp alignment mechanism, the aforementioned audio data streams, lip movement video data streams, and inertial sensor data streams are fused to construct an input structure for a multimodal neural network. This multimodal environment data structure maintains temporal synchronization and modal separation, allowing subsequent branch structures to process modal features separately.

[0125] The multimodal neural network architecture used has a main branch and auxiliary branches. The main branch is a deep convolution or temporal convolution module that focuses on extracting speech signal features from the audio data stream and outputs a basic noise reduction mask based on time-frequency mask modeling technology to identify areas in the spectrum that need to be suppressed or retained. The auxiliary branch extracts features from the inertial sensor data stream, identifies physical disturbance patterns occurring at the acquisition end, such as the systematic offset effects of non-speech factors such as bumps, movement, or collisions on the audio spectrum, and outputs spectrum offset compensation parameters for spectral correction.

[0126] The fusion operation uses a parameter-level weighting mechanism or an attention fusion mechanism to combine the basic noise reduction mask with the spectrum offset compensation parameters to construct noise suppression parameters that simultaneously take into account noise energy suppression and disturbance compensation capabilities. This fusion parameter is used to support the subsequent noise reduction processing stage. This fusion result has contextual relevance and environmental adaptability, significantly enhancing the system's robustness in complex non-static noise environments.

[0127] This embodiment utilizes a combined acquisition and modeling mechanism for audio, video, and inertial modalities to enable the system to identify and distinguish speech signals from non-speech disturbances in the environment. A spectral compensation mechanism for motion is introduced during the construction of the basic noise reduction mask, effectively addressing the spectrum offset issues of traditional single-modal noise reduction methods in dynamic scenarios. The noise suppression parameters generated by this fusion process possess enhanced time-frequency accuracy and environmental awareness, significantly reducing the impact of complex dynamic interference on speech clarity, providing a key support path for high-fidelity speech recognition in hearing-impaired assistance scenarios.

[0128] In one embodiment, the above step S30 includes:

[0129] S301, dividing the input information to be converted into multiple speech frames;

[0130] S302, performing spectrum analysis on each speech frame to generate speech spectrum features;

[0131] S303, processing the speech spectrum feature by using the basic noise reduction mask in the noise suppression parameter to obtain a processed spectrum feature;

[0132] S304, adjusting the processed spectrum characteristics by using the spectrum offset compensation parameter in the noise suppression parameter to obtain an adjusted spectrum characteristic;

[0133] S305, reconstructing the adjusted spectral features into a noise-reduced speech frame;

[0134] S306: Merge multiple noise-reduced speech frames into clear speech information.

[0135] In this embodiment, after obtaining the noise suppression parameters, in order to achieve fine-grained noise suppression on the input information to be converted, the original audio data is first divided along the time axis to obtain a series of short-duration speech frames. This speech frame division generally uses an overlapping window processing method with a fixed window length (e.g., 25ms) and a sliding step size (e.g., 10ms), which can balance time domain continuity and frequency domain stability.

[0136] Each frame of speech signal is converted into speech spectral features through spectral analysis techniques such as the Fast Fourier Transform (FFT) to reveal its frequency domain energy distribution pattern. These spectral features form a time-frequency spectrum, providing the data foundation for subsequent spectral masking.

[0137] During the spectral masking stage, the basic noise reduction mask in the noise suppression parameters is used to perform a point multiplication or weighting operation on the speech spectrum features frame by frame, thereby suppressing the background noise frequency band and enhancing the speech components. The basic noise reduction mask is typically a weight matrix that matches the dimensions of the speech spectrum. Its value range is between 0 and 1, and it is used to control the degree of energy preservation in different frequency bands.

[0138] To further compensate for spectral shifts caused by device disturbances or environmental non-stationarity, a spectrum offset compensation parameter is introduced. This parameter, trained based on inertial data, represents the shift adjustment relationship for the spectral energy distribution. Spectral offset compensation involves structurally adjusting the previously processed spectrum through methods such as interpolation repositioning and frequency band correction. This generates adjusted spectral features and restores speech spectrum distortion caused by disturbances.

[0139] Finally, the adjusted spectral features are restored to time-domain speech frames through signal reconstruction methods such as the inverse Fourier transform (iFFT). Then, through splicing operations such as overlap-addition, consecutive noise-reduced speech frames are synthesized into a complete speech sequence, ultimately outputting clear speech information. This output retains valid speech components, suppresses background interference, and corrects for external disturbances.

[0140] This embodiment uses a spectral domain step-by-step processing method to apply the multi-branch information in the noise suppression parameters in stages to the speech spectrum feature processing process, thereby improving the accuracy and adaptability of noise suppression. The spectrum analysis and processing at the speech frame level ensures the balance of the system in terms of spectrum and time resolution, and the spectrum offset compensation mechanism significantly reduces the speech distortion caused by device displacement, hand shaking, etc. The clear speech information finally generated has higher language intelligibility and speech naturalness, which builds a stable and high-quality input foundation for the subsequent recognition stage, and is particularly suitable for the auxiliary recognition needs of hearing-impaired people in barrier-free voice communication scenarios in noisy environments.

[0141] In one embodiment, the above step S50 includes:

[0142] S501, extracting time-frequency spectrum features of the clear speech information;

[0143] S502, identifying three-dimensional mouth shape key points in the lip movement video data;

[0144] S503, aligning the time-frequency spectrum features with the time axis of the three-dimensional mouth shape key points to generate an alignment result;

[0145] S504, generating lip movement compensation data based on the alignment result;

[0146] S505 , detecting easily confused phonemes in the initial text information, and correcting the easily confused phonemes using the lip movement compensation data to generate corrected target text information.

[0147] In this embodiment, to further improve the accuracy of speech recognition, especially in scenarios with complex noise or ambiguous pronunciation, it is necessary to coordinate and compensate for auditory and visual information. First, time-frequency features are extracted from the clear speech information. This process uses short-time Fourier transform to obtain the energy distribution of the audio signal in both time and frequency dimensions, and constructs a time-frequency graph as a high-dimensional representation of the sound expression. This representation retains the formant trajectory and energy mutation information in the clear speech, making it suitable for corresponding analysis with visual features.

[0148] At the same time, by identifying key points of the 3D mouth shape in lip movement video data, a convolutional neural network and a 3D face reconstruction algorithm can be used to detect the lip area and extract the 3D temporal coordinate information of feature points such as the mouth corners, lip lines, and upper and lower lip contours. This information reflects the dynamic trajectory of the lips during pronunciation, providing important clues for identifying phoneme pronunciation.

[0149] Next, the time-frequency spectrum features are aligned with the time axis of the 3D lip key points. This alignment strategy, based on a frame-level timestamp matching mechanism, combines the sampling frequency with the video frame rate to establish a one-to-one mapping relationship, ensuring that the audio frames and lip frames are synchronized in time. Based on this alignment, an attention mechanism or graph neural network is used to generate lip compensation data, which encodes the structural correspondence between the lip trajectory and the audio changes.

[0150] The system then detects easily confused phonemes in the initial text, including voiced and unvoiced pairs, tongue position approximations, and aspirated and unaspirated phoneme pairs. This is typically achieved through attention analysis based on the phoneme sequence or acoustic model confidence lower bound screening. After acquiring lip movement compensation data, these easily confused phonemes are corrected based on the consistency between mouth shape changes and audio features. By replacing or adjusting the pronunciation annotation method, these phonemes are corrected to higher confidence levels, ultimately generating target text with more consistent semantics and pronunciation. This target text retains semantic integrity while correcting pronunciation errors in audio recognition.

[0151] This embodiment achieves multimodal alignment by combining clear voice information with the dynamic features of lip movement video data, and compensates for recognition uncertainty in the auditory modality through lip trajectory analysis, effectively solving the problem of phoneme confusion and misrecognition under noise interference. It improves the robustness and resolution of recognition, and is particularly suitable for practical application scenarios with diverse accents, uneven speaking speeds, or unclear pronunciation of high-frequency phonemes. The corrected target text information has higher speech and semantic consistency, effectively improving the recognition accuracy of hearing-impaired people when communicating through voice input systems, and providing higher-quality text output support for barrier-free interaction.

[0152] In one embodiment, the above step S60 includes:

[0153] S601, obtaining the current user identity;

[0154] S602, querying a pre-built speaker style vector database according to the current user identity;

[0155] S603, extracting a speaker style vector corresponding to the current user identity from the speaker style vector database;

[0156] S604: Verify the validity of the speaker style vector, and load the verified speaker style vector.

[0157] In this embodiment, when converting text information into voice information, a speaker style vector is introduced as an expression control condition to enhance the personalized expression of the voice output. First, the user identity of the current interaction needs to be identified. This identity can be obtained through user account information bound to the front-end system, hardware device fingerprint, communication channel token, or an identification field explicitly provided in the session context. This ensures the traceability and uniqueness of the user's voice style data.

[0158] Based on the user's ID, the system accesses a pre-built database of speaker style vectors. This database is constructed using clustering modeling, with each vector corresponding to a different combination of timbre feature dimensions, such as pitch baseline, speech rate pattern, resonance distribution, and emotional encoding. Databases typically employ vectorized indexing structures for efficient retrieval, such as inverted indexes or cosine similarity indexing to improve query response.

[0159] After retrieving a speaker style vector matching the user's ID from the database, the vector's validity must be verified to ensure system stability and speech generation consistency. This verification process includes checking the vector's dimensionality, whether its numerical range complies with standard model inputs, whether its timestamp falls within the acceptable update cycle, and whether it matches the current task model version. For vectors that fail verification, fallback mechanisms can be triggered, such as loading a default style or guiding the user through style retraining.

[0160] Ultimately, the verified speaker style vector is loaded into the subsequent text-to-speech model call path, acting as a control condition in the speech signal generation process. The entire processing chain technically ensures user perception stability and model input consistency, ensuring the system's ability to support personalized speech synthesis for multiple users and in multiple contexts.

[0161] This embodiment achieves automated and personalized configuration of style control in speech synthesis by introducing a user identity recognition mechanism during the text input phase and extracting matching speaker style vectors from a structured database based on this identity. This mechanism lowers the threshold for users to repeatedly train the model, avoids the reliance on a large number of voice samples in traditional TTS models, and can still achieve high-quality, user-specific speech generation output in scenarios with limited data resources or limited user capabilities. It is particularly suitable for hearing-impaired people to express their emotional tone or personality preferences through text input systems, enhancing the intimacy and expressiveness of auditory interaction, and providing an important foundation for building an accessible information environment.

[0162] In one embodiment, the above step S70 includes:

[0163] S701, parsing the input information to be converted into a phoneme sequence;

[0164] S702, injecting the speaker style vector into the style control layer of the text-to-speech model;

[0165] S703, modulating the rhythmic features of the phoneme sequence through the style control layer;

[0166] S704, inputting the modulated phoneme sequence into the acoustic model, and generating acoustic features through the acoustic model;

[0167] S705: Input the acoustic features into a neural vocoder, and synthesize the target speech information through the neural vocoder.

[0168] In this embodiment, after completing text input, in order to achieve speech synthesis with a personalized voice style, the input information to be converted needs to be structured and processed, and the entire speech generation path is driven by the pre-loaded speaker style vector. The first step is to parse the text input into a phoneme sequence. A phoneme sequence is the smallest set of speech units for language expression. The text parsing module uses a phoneme annotation rule set, such as the International Phonetic Alphabet (IPA) or a language-specific phoneme system, to generate a phoneme stream through lexical analysis, spelling standardization, and pronunciation rule mapping.

[0169] The extracted speaker style vector is then injected into the style control layer of the text-to-speech model. This layer, typically located between front-end feature extraction and acoustic modeling, adjusts the input phoneme's pronunciation, rhythm, intonation, pitch, and other prosodic features. This injection can be done through vector concatenation, gated modulation, or conditional normalization, effectively influencing the subsequent prosodic parameter generation process.

[0170] The modulated phoneme sequence is input into the acoustic model for feature prediction. The acoustic model can be an autoregressive model based on the Transformer structure or a non-autoregressive structure based on the diffusion model. Its task is to map the language unit represented by the symbol into a continuous set of acoustic feature parameters at the frame level, including Mel spectrum, fundamental frequency curve and sound intensity distribution, etc., to ensure the physical feasibility of speech signal generation.

[0171] These acoustic features are then fed into a neural vocoder module, which decodes continuous acoustic parameters into a time-domain audio waveform. Neural vocoders such as HiFi-GAN, WaveGlow, or Parallel WaveGAN, through trained and optimized deconvolution and gating mechanisms, can maintain high-fidelity reproduction of speech details, achieving natural and smooth speech output.

[0172] Throughout the entire process, the injection of the speaker style vector ensures that the synthesized speech not only accurately expresses the text content, but also presents the timbre, rhythm and emotional attitude that matches the user's identity characteristics, completing the personalized customization of the speech output.

[0173] This embodiment introduces a speaker style vector and injects it into the text-to-speech model to control style expression. This not only achieves text-to-speech conversion, but also imbues the output speech with highly personalized linguistic prosodic features. This significantly improves the diversity and auditory naturalness of speech expression, demonstrating greater adaptability for the hearing-impaired, speech-impaired, and those who need personalized voice interaction to express their intentions. This approach effectively reduces synthesis latency while ensuring voice quality, enhancing the real-time nature and exclusivity of voice interaction.

[0174] In one embodiment, the above step S80 includes:

[0175] S801, analyzing phoneme boundaries of the target speech information, and generating lip movement parameters according to the phoneme boundaries;

[0176] S802, generating lip movement synchronization information based on the lip movement parameters;

[0177] S803, extracting the energy envelope of the target voice information, and mapping the energy envelope into tactile vibration parameters;

[0178] S804: Generate tactile feedback information according to the tactile vibration parameter.

[0179] In this embodiment, in order to enhance the multimodal synchronous expression of the speech output process, it is necessary to convert the target speech information into other perceptible forms of data output. The first step is to analyze the phoneme boundaries in the target speech information. The phoneme boundaries represent the start and end points of each pronunciation unit on the time axis, which can be obtained by parsing the acoustic features through the time alignment tags recorded in the acoustic model or the external alignment model (such as a forced aligner). The accuracy of the phoneme boundaries directly affects the accuracy of lip synchronization and is the key basis for establishing the correspondence between speech and vision.

[0180] Based on the phoneme boundaries, lip motion parameters are further generated. Lip motion parameters refer to the lip morphology data corresponding to a specific phoneme, such as mouth opening width, lip roundness, and upper and lower lip distance. They are often expressed as three-dimensional spatial coordinates or motion control vectors. These parameters can be obtained by looking up a speech pronunciation mapping table based on the phoneme category, or they can be directly predicted by a trained speech-to-lip regression model. The generated lip motion parameters are aligned along the phoneme boundaries in the time dimension, forming a continuously changing motion trajectory.

[0181] The aforementioned lip motion parameters are converted into lip synchronization information, namely a lip synchronization control sequence. This control sequence can be used to drive 3D virtual human facial expression animation, speech recognition assistance visualization modules, or embedded visual prompt devices. Lip synchronization information must be synchronized with the target speech information at the frame level to avoid misunderstanding caused by mismatch between speech and lip shape. This is particularly important in silent assistance or hearing-impaired assistance scenarios.

[0182] The target speech information is further processed to extract the energy envelope. The energy envelope describes the intensity fluctuation trend of the audio signal in the time dimension and is typically extracted through short-time Fourier transform, frame energy statistics, or time-direction convolution of the Mel-spectrogram energy map. The extracted energy envelope is modulated based on dimensions such as frequency intensity changes, rhythm density, and burst peaks, and converted into tactile vibration parameters such as vibration intensity, frequency, and duration. This parameter mapping can be achieved through a lookup table, nonlinear mapping function, or neural network regression model.

[0183] Finally, the aforementioned tactile vibration parameters are used to drive a tactile feedback device to generate tactile feedback information. This tactile feedback device can be a wrist-worn vibration module, a piezoelectric vibrator in the screen frame, or a dedicated acoustic feedback terminal. The generated feedback information must be precisely synchronized and presented in real time during the playback of the target voice message, ensuring that users can perceive the rhythm and emotional changes of the voice through tactile perception in scenarios where auditory channels are lacking, thereby enhancing the accessibility of the interaction and the integrity of the perceived expression.

[0184] After tactile feedback information is generated, it is typically distributed synchronously to a terminal component with tactile output capabilities to provide real-time feedback that the user can perceive. This tactile output component can be a module equipped with a linear resonant actuator (LRA), a piezoelectric actuator, a vibration motor, or a multi-point vibration array. Depending on the actual hardware configuration, its drive method, vibration response characteristics, and control parameter interface may vary. To achieve uniform adaptation for different devices, after generating tactile feedback information, the system must map this information into a standardized vibration control instruction set, which includes parameters such as vibration frequency, amplitude, duration, and activation timing. The encoding of these control instructions can be modulated based on a vibration intensity level table or a perceptual sensitivity function, so that the output rhythm of tactile feedback during speech playback is coordinated with the fluctuations in speech energy. These standardized instructions are then pushed to the target device in real time via the tactile control interface. If the system is deployed in a multimodal interactive terminal, it must also determine the current user interaction state to ensure that tactile feedback does not conflict with visual or audio output. For example, in silent mode or headphone mode, the tactile channel can be prioritized to enhance information perception. To synchronize tactile output with the target speech information, the system must maintain a synchronization timing reference line. This reference line is derived by aligning the phoneme timing annotation results in the speech synthesis module with the actual playback timeline. Upon receiving control commands, the haptic drive system must reference this timing reference to implement microsecond or millisecond delay control to ensure the coordination of feedback from different modalities. In edge device or mobile deployment scenarios, this process also requires lightweight processing, such as through interrupt-driven scheduling or low-power vibration command compression mechanisms, to reduce system resource usage and extend battery life. Finally, to ensure the output quality and consistency of tactile feedback, a feedback verification mechanism can be introduced. For example, the drive response signal transmitted back from the device or the actual vibration intensity captured by the inertial measurement unit (IMU) can be compared with preset target parameters to adjust subsequent output strategies in real time, further enhancing the consistency and naturalness of the user's tactile experience.

[0185] After generating tactile feedback information, in order to further optimize the response speed and resource utilization efficiency of the overall interaction process, it is necessary to introduce a cross-computing platform execution scheduling mechanism to dynamically determine the execution location of subsequent processing tasks. The decision-making basis for resource scheduling includes the current network connection status, the computing power of the terminal device, and the model complexity of the task to be processed. The inputs involved in the scheduling process include network link quality indicators (such as throughput, packet loss rate, RTT), real-time load information on the edge device (such as CPU utilization, power consumption threshold), and the predicted value of the computing resource demand of the current module to be executed. To adapt to the changing network environment, the system implements the allocation strategy of computing resources between edge devices and the cloud through a dynamic load balancer (DLB). Built on the monitoring subsystem, the DLB collects network and device status data in real time, calculates the supportable data stream transmission scale based on the bandwidth estimation model, and makes task scheduling decisions based on the real-time requirements of the current module. The scheduling algorithm adopts a segmented judgment strategy: when the network bandwidth is greater than a specific threshold and the latency is stable, the low-latency mode is enabled, and the execution path prioritizes the lightweight model version deployed on the edge side for full-process local inference; when the network quality fluctuates or the bandwidth is insufficient, the system switches to the joint computing mode, and only after the edge device preprocesses and compresses the data features, it is transmitted to the cloud through the WebRTC protocol for inference.

[0186] In order to control the data overhead of compressed transmission, the data output from the edge side is encoded using sparsification and quantization. Specifically, the FP16 half-precision floating-point format is used to uniformly compress the acoustic features and lip movement parameters, and the dynamic axis sparsity strategy is further adopted for the sequence dimension to remove the vector channels of low-weight nodes without affecting the integrity of the information expression. This compression method significantly reduces the communication load and improves the response capability in a weak network environment. At the same time, the system introduces an execution delay evaluation module, using the total delay from the start of the task to the completion of the feedback as a closed-loop feedback variable, which is used to dynamically adjust the subsequent task allocation strategy, so that the scheduling decision has historical response perception and adaptive adjustment capabilities.

[0187] Example: In a bank branch counter service scenario, a hearing-impaired customer comes to inquire about financial products. Traditionally, due to speech barriers, these customers often rely on pen and paper or sign language, severely impacting interaction efficiency and privacy. At this point, a real-time bidirectional speech-to-text conversion system deployed on the counter terminal device comes into play.

[0188] First, the system receives voice input from the bank's account manager, automatically identifies it as a voice message, and simultaneously activates the audio, video, and inertial sensor acquisition channels. The audio channel records the account manager's speech, the video channel captures their facial and lip movements, and the inertial sensor channel assists in analyzing background motion interference. These three data streams are integrated to form multimodal environmental data. A multimodal neural network is then used to extract basic noise reduction masks and spectrum offset compensation parameters, which are then fused to generate noise suppression parameters.

[0189] The system then divides the received voice input into frames and performs spectral analysis. Using the generated noise suppression parameters, it performs multi-stage noise reduction and spectral correction on the spectral features, ultimately reconstructing a continuous and clear voice signal. The clear voice information is then input into the speech recognition model to output the initial text results. Considering that the account manager's speech may contain unclear pronunciation or excessive speaking speed, which may lead to misrecognition, the system further combines the three-dimensional mouth shape key points extracted from the lip movement video with the time-frequency spectrum features of the clear speech for time alignment. By generating lip movement compensation data, it corrects the easily confused phonemes in the initial text, resulting in semantically accurate target text information.

[0190] The target text information is presented to hearing-impaired customers in real time through an accessible assistive screen, and customers can respond directly through the text input interface after reading. The system recognizes the input type as text information, automatically obtains the user identity (such as customer ID) and calls the corresponding pre-trained speaker style vector. In the text-to-speech module, the input text is first parsed into a phoneme sequence, and its rhythmic features are modulated through the style control layer. Then, the acoustic model and neural vocoder are used to synthesize the target speech information with the customer's specific voice style.

[0191] To enhance natural interaction, the system generates lip-sync information aligned with the speech output, driving the virtual lip animation on the auxiliary screen at the counter. It also extracts the energy envelope from the speech signal to generate corresponding tactile feedback, which drives wearable devices (such as smart wristbands) to provide rhythmic vibration reminders. The entire two-way interaction process is kept to a latency of less than 200ms, enabling hearing-impaired customers to perceive the speech content and feedback rhythm in real time, significantly improving their autonomous communication skills and service experience in financial services.

[0192] Similarly, a hearing-impaired patient arrives at the guidance service area of ​​a large hospital and wishes to learn about department queues and specialist clinic information. The smart guidance screens and mobile devices deployed on-site are equipped with the real-time, bidirectional speech-to-text conversion system supported by this method.

[0193] When the nurse guide interacts with the patient using voice, the system identifies this as voice input and activates multi-channel acquisition mode, collecting the nurse's voice, mouth shape, and surrounding dynamics. This fused multimodal environmental data is fed into a neural network module to extract noise reduction masks and compensation parameters, effectively addressing the complex background sounds in the outpatient hall, such as announcements and the clamor of people.

[0194] The system then performs fine-grained noise reduction on the speech data through spectral analysis and frequency-domain modulation, obtaining clear speech information for recognition. Because medical terms can have similar pronunciations (such as "gastroscope" and "laryngoscopy"), the system incorporates a lip movement compensation mechanism to correct for confusing phonemes in the recognized text, ensuring the consistency and accuracy of the output target text information in terms of medical semantics.

[0195] When a patient types a text response through the system's input interface, such as specifying "Need a gastroenterologist," the system automatically recognizes the input as text and retrieves the speaker style vector associated with the user's identity, such as the patient's usual soft intonation template. This style vector is injected into the speech synthesis pipeline to ensure the output speech matches the patient's typical style. The speech is then played over the waiting area speakers to alert medical staff or serve as a guide for further interaction.

[0196] At the same time, to ensure that hearing-impaired patients can perceive the real-time and output rhythm of the current interaction, the system maps the phoneme boundaries corresponding to the voice content into lip animation driving parameters, and presents them in the interactive virtual human interface on the side of the guidance screen; the energy characteristics of the voice will also be converted into vibration patterns, and the vibration rhythm will be emitted through the vibration prompter worn by the patient to guide them to pay attention to content changes or input confirmation.

[0197] The entire operating process not only meets the immediate needs of hearing-impaired users in medical facilities, but also avoids the inconvenience of relying on visual cues in traditional queuing systems while maintaining privacy, accuracy, and comfort. It is especially suitable for the medical assistance needs of elderly hearing-impaired patients.

[0198] This embodiment combines the phoneme boundary analysis in the target voice information with the lip movement parameters to achieve real-time generation of lip synchronization information, effectively supporting the visual expression of voice content and improving the language comprehension efficiency of hearing-impaired users. At the same time, by extracting the energy envelope of the voice signal and mapping it into adjustable tactile vibration parameters, it can provide real-time sensory feedback in silent or high-noise scenes, enhancing the user's perception of voice rhythm and emotional changes. It is not only highly practical in multimodal barrier-free interaction, but also demonstrates wide adaptability and scalability in wearable assistive devices and immersive interactive systems.

[0199] In one embodiment, a device for bidirectional conversion of speech to text is provided, which corresponds to the method for bidirectional conversion of speech to text in the above embodiment. Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the speech-to-text bidirectional conversion device of the present invention. It includes an input type recognition module 10, a multimodal perception module 20, a speech noise reduction module 30, a speech recognition module 40, a lip movement compensation module 50, a style vector extraction module 60, a speech synthesis module 70, and a feedback generation module 80. Each functional module is described in detail below:

[0200] An input type identification module 10 is configured to receive input information to be converted and determine the type of the input information;

[0201] a multimodal perception module 20 for acquiring multimodal environment data including lip movement video data when the input information type is voice information, and generating noise suppression parameters based on the multimodal environment data;

[0202] A speech noise reduction module 30 is configured to perform noise reduction processing on the input information to be converted according to noise suppression parameters to generate clear speech information;

[0203] A speech recognition module 40, configured to input the clear speech information into a speech recognition model and output initial text information;

[0204] a lip movement compensation module 50 for generating lip movement compensation data based on the clear voice information and the lip movement video data, and using the lip movement compensation data to correct the initial text information to generate target text information;

[0205] A style vector extraction module 60 is configured to obtain a pre-generated speaker style vector when the input information type is text information;

[0206] a speech synthesis module 70 for inputting the input information to be converted into text into a speech model and referencing the speaker style vector to output target speech information;

[0207] The feedback generation module 80 is configured to generate lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information.

[0208] In one embodiment, the multimodal perception module 20 is specifically configured to:

[0209] When the input information type is voice information, start the audio acquisition channel, video acquisition channel and inertial sensor acquisition channel;

[0210] Acquire audio data streams through the audio acquisition channel, acquire lip movement video data streams through the video acquisition channel, and acquire inertial sensor data streams through the inertial sensor acquisition channel;

[0211] Integrating the audio data stream, the lip movement video data stream, and the inertial sensor data stream to generate multimodal environmental data;

[0212] Inputting the multimodal environmental data into a multimodal neural network;

[0213] Processing an audio data stream in the multimodal environmental data through a main branch of the multimodal neural network to generate a basic noise reduction mask;

[0214] Processing an inertial sensor data stream in the multimodal environmental data through an auxiliary branch of the multimodal neural network to generate a spectrum shift compensation parameter;

[0215] The basic noise reduction mask and the spectrum shift compensation parameter are combined to generate noise suppression parameters.

[0216] In one embodiment, the speech noise reduction module 30 is specifically configured to:

[0217] Dividing the input information to be converted into a plurality of speech frames;

[0218] Perform spectrum analysis on each speech frame to generate speech spectrum features;

[0219] Processing the speech spectrum feature by using the basic noise reduction mask in the noise suppression parameter to obtain a processed spectrum feature;

[0220] Adjusting the processed spectrum characteristics by using the spectrum offset compensation parameter in the noise suppression parameter to obtain an adjusted spectrum characteristic;

[0221] Reconstructing the adjusted spectral features into a noise-reduced speech frame;

[0222] Merge multiple noise-reduced speech frames into clear speech information.

[0223] In one embodiment, the lip movement compensation module 50 is specifically configured to:

[0224] Extracting time-frequency spectrum features of the clear speech information;

[0225] Identifying three-dimensional mouth shape key points in the lip movement video data;

[0226] Aligning the time-frequency spectrum features with the time axis of the three-dimensional mouth shape key points to generate an alignment result;

[0227] generating lip movement compensation data based on the alignment result;

[0228] Confusing phonemes in the initial text information are detected, and the confusing phonemes are corrected using the lip movement compensation data to generate corrected target text information.

[0229] In one embodiment, the style vector extraction module 60 is specifically configured to:

[0230] Get the current user identity;

[0231] Querying a pre-built speaker style vector database according to the current user identity;

[0232] Extracting a speaker style vector corresponding to the current user identity from the speaker style vector database;

[0233] Verify the validity of the speaker style vector and load the verified speaker style vector.

[0234] In one embodiment, the speech synthesis module 70 is specifically configured to:

[0235] Parsing the input information to be converted into a phoneme sequence;

[0236] injecting the speaker style vector into the style control layer of the text-to-speech model;

[0237] modulating the prosodic features of the phoneme sequence through the style control layer;

[0238] Input the modulated phoneme sequence into the acoustic model and generate acoustic features through the acoustic model;

[0239] The acoustic features are input into a neural vocoder, and the target speech information is synthesized by the neural vocoder.

[0240] In one embodiment, the feedback generation module 80 is specifically configured to:

[0241] Analyzing phoneme boundaries of the target speech information and generating lip movement parameters according to the phoneme boundaries;

[0242] generating lip synchronization information based on the lip motion parameters;

[0243] extracting an energy envelope of the target speech information, and mapping the energy envelope into tactile vibration parameters;

[0244] Tactile feedback information is generated according to the tactile vibration parameter.

[0245] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a method for bidirectional conversion of speech to text.

[0246] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a method for bidirectional conversion of speech to text.

[0247] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0248] receiving input information to be converted and determining the type of the input information;

[0249] When the input information type is voice information, obtaining multimodal environment data including lip movement video data, and generating noise suppression parameters based on the multimodal environment data;

[0250] Performing noise reduction processing on the input information to be converted according to the noise suppression parameters to generate clear speech information;

[0251] Inputting the clear speech information into a speech recognition model and outputting initial text information;

[0252] generating lip movement compensation data based on the clear voice information and the lip movement video data, and correcting the initial text information using the lip movement compensation data to generate target text information;

[0253] When the input information type is text information, obtaining a pre-generated speaker style vector;

[0254] Inputting the input information to be converted into a text-to-speech model and referencing the speaker style vector, and outputting target speech information;

[0255] Lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information are generated.

[0256] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0257] receiving input information to be converted and determining the type of the input information;

[0258] When the input information type is voice information, obtaining multimodal environment data including lip movement video data, and generating noise suppression parameters based on the multimodal environment data;

[0259] Performing noise reduction processing on the input information to be converted according to the noise suppression parameters to generate clear speech information;

[0260] Inputting the clear speech information into a speech recognition model and outputting initial text information;

[0261] generating lip movement compensation data based on the clear voice information and the lip movement video data, and correcting the initial text information using the lip movement compensation data to generate target text information;

[0262] When the input information type is text information, obtaining a pre-generated speaker style vector;

[0263] Inputting the input information to be converted into a text-to-speech model and referencing the speaker style vector, and outputting target speech information;

[0264] Lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information are generated.

[0265] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0266] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0267] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0268] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for bidirectional conversion of speech to text, characterized in that: The following steps are involved: receiving input information to be converted and determining the type of the input information; When the input information type is voice information, obtaining multimodal environment data including lip movement video data, and generating noise suppression parameters based on the multimodal environment data; Performing noise reduction processing on the input information to be converted according to the noise suppression parameters to generate clear speech information; Inputting the clear speech information into a speech recognition model and outputting initial text information; generating lip movement compensation data based on the clear voice information and the lip movement video data, and correcting the initial text information using the lip movement compensation data to generate target text information; When the input information type is text information, obtaining a pre-generated speaker style vector; Inputting the input information to be converted into a text-to-speech model and referencing the speaker style vector, and outputting target speech information; Lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information are generated.

2. The method for bidirectional conversion of speech to text according to claim 1, wherein: When the input information type is voice information, obtaining multimodal environment data including lip movement video data, and generating noise suppression parameters based on the multimodal environment data, including: When the input information type is voice information, start the audio acquisition channel, video acquisition channel and inertial sensor acquisition channel; Acquire audio data streams through the audio acquisition channel, acquire lip movement video data streams through the video acquisition channel, and acquire inertial sensor data streams through the inertial sensor acquisition channel; Integrating the audio data stream, the lip movement video data stream, and the inertial sensor data stream to generate multimodal environmental data; Inputting the multimodal environmental data into a multimodal neural network; Processing an audio data stream in the multimodal environmental data through a main branch of the multimodal neural network to generate a basic noise reduction mask; Processing an inertial sensor data stream in the multimodal environmental data through an auxiliary branch of the multimodal neural network to generate a spectrum shift compensation parameter; The basic noise reduction mask and the spectrum shift compensation parameter are combined to generate noise suppression parameters.

3. The method for bidirectional conversion of speech to text according to claim 1, wherein: Performing noise reduction processing on the input information to be converted according to the noise suppression parameter to generate clear speech information, including: Dividing the input information to be converted into a plurality of speech frames; Perform spectrum analysis on each speech frame to generate speech spectrum features; Processing the speech spectrum feature by using the basic noise reduction mask in the noise suppression parameter to obtain a processed spectrum feature; Adjusting the processed spectrum characteristics by using the spectrum offset compensation parameter in the noise suppression parameter to obtain an adjusted spectrum characteristic; Reconstructing the adjusted spectral features into a noise-reduced speech frame; Merge multiple noise-reduced speech frames into clear speech information.

4. The method for bidirectional conversion of speech to text according to claim 1, wherein: Generating lip movement compensation data based on the clear voice information and the lip movement video data, and correcting the initial text information using the lip movement compensation data to generate target text information, including: Extracting time-frequency spectrum features of the clear speech information; Identifying three-dimensional mouth shape key points in the lip movement video data; Aligning the time-frequency spectrum features with the time axis of the three-dimensional mouth shape key points to generate an alignment result; generating lip movement compensation data based on the alignment result; Confusing phonemes in the initial text information are detected, and the confusing phonemes are corrected using the lip movement compensation data to generate corrected target text information.

5. The method for bidirectional conversion of speech to text according to claim 1, wherein: When the input information type is text information, obtaining a pre-generated speaker style vector includes: Get the current user identity; Querying a pre-built speaker style vector database according to the current user identity; Extracting a speaker style vector corresponding to the current user identity from the speaker style vector database; Verify the validity of the speaker style vector and load the verified speaker style vector.

6. The method for bidirectional conversion of speech to text according to claim 1, wherein: Inputting the input information to be converted into a text-to-speech model and referencing the speaker style vector to output target speech information includes: Parsing the input information to be converted into a phoneme sequence; injecting the speaker style vector into the style control layer of the text-to-speech model; modulating the prosodic features of the phoneme sequence through the style control layer; Input the modulated phoneme sequence into the acoustic model and generate acoustic features through the acoustic model; The acoustic features are input into a neural vocoder, and the target speech information is synthesized by the neural vocoder.

7. The method for bidirectional conversion of speech to text according to claim 1, wherein: Generating lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information, including: Analyzing phoneme boundaries of the target speech information and generating lip movement parameters according to the phoneme boundaries; generating lip synchronization information based on the lip motion parameters; extracting an energy envelope of the target speech information, and mapping the energy envelope into tactile vibration parameters; Tactile feedback information is generated according to the tactile vibration parameter.

8. A device for bidirectional conversion of speech and text, characterized in that: The speech-to-text bidirectional conversion device comprises: An input type identification module, configured to receive input information to be converted and determine the type of the input information; a multimodal perception module, configured to, when the input information type is voice information, obtain multimodal environment data including lip movement video data, and generate noise suppression parameters based on the multimodal environment data; A speech noise reduction module, configured to perform noise reduction processing on the input information to be converted according to noise suppression parameters to generate clear speech information; A speech recognition module, configured to input the clear speech information into a speech recognition model and output initial text information; a lip movement compensation module, configured to generate lip movement compensation data based on the clear voice information and the lip movement video data, and use the lip movement compensation data to correct the initial text information to generate target text information; A style vector extraction module, configured to obtain a pre-generated speaker style vector when the input information type is text information; a speech synthesis module, configured to input the input information to be converted into a text-to-speech model and reference the speaker style vector to output target speech information; The feedback generation module is used to generate lip synchronization information synchronized with the target voice information and tactile feedback information corresponding to the energy characteristics of the target voice information.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a speech-to-text bidirectional conversion program stored in the memory and capable of running on the processor. When the speech-to-text bidirectional conversion program is executed by the processor, the steps of the speech-to-text bidirectional conversion method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a speech-to-text bidirectional conversion program, which, when executed by a processor, implements the steps of the speech-to-text bidirectional conversion method according to any one of claims 1 to 7.

Images