Speech translation method and system based on artificial intelligence

Abstract

The invention relates to the technical field of speech translation, in particular to a speech translation method and system based on artificial intelligence, and the method comprises the steps: obtaining a normalized speech waveform and a lip image sequence, and sequentially inputting the normalized speech waveform and the lip image sequence into a front-end network and a rear-end network; extracting the first audio feature sequence, and obtaining a second audio feature sequence through one-dimensional convolution dimensionality reduction and time sequence modeling; processing the lip image sequence to obtain a first visual feature sequence, and performing one-dimensional convolution dimension reduction on the obtained first visual feature sequence to obtain a second visual feature sequence; establishing a one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence by using a multi-head key value memory network to obtain an enhanced visual feature sequence, and fusing the enhanced visual feature sequence with the second audio feature sequence in a joint cross-modal fusion module to form a fused feature sequence; and normalizing the character sequence to obtain a source language text sequence, and inputting the source language text sequence into a neural machine translation model for semantic mapping to generate a target language text sequence. According to the invention, the problem of poor translation effect caused by poor speech recognition effect in a noise environment can be solved.

Description

Technical Field

[0001] This invention relates to the field of speech translation technology, and in particular to a speech translation method and system based on artificial intelligence. Background Technology

[0002] With the development of smart classrooms, online teaching, multinational conferences, and automatic recording systems, the demand for real-time or near-real-time automatic translation and subtitling of spoken content such as classroom lectures and conference speeches is becoming increasingly prominent. Existing speech translation systems typically employ a cascaded approach of "ASR (Automatic Speech Recognition) + Machine Translation (MT): first converting speech into source language text, and then translating the source language text into target language text. However, in real-world scenarios such as classrooms or conferences, issues such as multiple speakers, environmental noise, echoes, and reverberation often arise, leading to omissions, misspellings, missing punctuation, and non-standard text in the recognition results. These errors are further propagated and amplified in the cascaded system, causing semantic deviations and poor readability in the translation results, making it difficult to meet practical application needs.

[0003] Furthermore, existing technologies still primarily focus on improving the accuracy of speech-to-text recognition, paying insufficient attention to the text structure and semantic expression relied upon in the translation stage. For example, there is a lack of a unified mechanism for standardized processing of sentence segmentation, punctuation restoration, and numbers / time / units, making the output text unfavorable for semantic mapping by neural machine translation models. At the same time, some methods lack effective constraints on intermediate representations, which can easily lead to problems such as insufficient utilization of visual features and inadequate audiovisual coordination, and may still cause a decline in translation quality under varying noise conditions or in scenarios with dense technical terms. Summary of the Invention

[0004] In view of the above technical problems, this invention provides an artificial intelligence-based speech translation method and system, aiming to solve the problems of easy propagation and amplification of recognition errors in existing speech recognition and machine translation cascade schemes in noisy environments such as classrooms and meetings, and semantic deviation and poor readability caused by the lack of punctuation and non-standard expression in text sequences. By introducing lip-sync video and using a multi-head key-value memory network to compensate and enhance visual features, and combining cross-modal fine fusion to improve the accuracy and robustness of source language text sequence acquisition, the invention further processes the text sequence by sentence segmentation, punctuation restoration and text normalization before inputting it into a neural machine translation model for semantic mapping, generating and outputting a target language text sequence corresponding to the source language text sequence, thereby improving the translation quality and stability in noisy scenarios.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of the present invention, an artificial intelligence-based speech translation method is proposed, characterized in that the method includes: Acquire audiovisual samples containing the target speaker's speech signal and a synchronized lip video sequence; preprocess the audiovisual samples to obtain a normalized speech waveform and a corresponding lip image sequence. The normalized speech waveform is input into the audio front-end network, where temporal convolution and self-supervised feature extraction are performed to obtain the first audio feature sequence. The first audio feature sequence is then input into the audio back-end network, where dimensionality reduction is achieved through one-dimensional convolution and temporal modeling to obtain the second audio feature sequence used for speech content representation. The lip image sequence is input into the visual front-end network, and a first visual feature sequence is obtained through three-dimensional convolution and spatial feature extraction; the first visual feature sequence is input into the visual back-end network, and a second visual feature sequence is obtained through one-dimensional convolution dimensionality reduction. A one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence is established using a multi-head key-value memory network to obtain an enhanced visual feature sequence; The enhanced visual feature sequence and the second audio feature sequence are input into the joint cross-modal fusion module for fusion to obtain the fused feature sequence; The fused feature sequence is input into the encoder network for temporal encoding to obtain the encoded feature sequence. The encoded feature sequence is then input into the sequence decoding branch and the connection temporal classification decoding branch, respectively. The first character probability sequence output by the sequence-to-sequence decoding branch and the second character probability sequence output by the connection temporal classification decoding branch are used to generate the text sequence corresponding to the audiovisual sample. The text sequence is segmented, punctuation restored, and text normalized to obtain a source language text sequence. The source language text sequence is then input into a neural machine translation model for semantic mapping to generate a target language text sequence corresponding to the source language text sequence, and the target language text sequence is output.

[0007] Furthermore, acquiring the lip image sequence includes: By performing face detection and facial key point detection on the audiovisual samples, the speaker's mouth region is located, and a fixed-size image block is cropped with the mouth region as the center as the lip image sequence. The lip image sequence is converted into a grayscale image and normalized according to the overall mean and variance of all the lip image sequences. When acquiring the second visual feature sequence, the following are included: In the visual front-end network, the lip image sequence is spatiotemporally convolved by a three-dimensional convolutional layer to extract local spatiotemporal features, and deep spatial features are extracted by a convolutional neural network with residual blocks as basic units to obtain the first visual feature sequence. In the visual backend network, the feature dimension of the first visual feature sequence is compressed by a one-dimensional convolutional layer to obtain the second visual feature sequence.

[0008] Furthermore, in establishing the one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence, the following steps are included: The second visual feature sequence is written into the key memory unit of the multi-head key-value memory network, and the second audio feature sequence is written into the corresponding value memory unit. During the query, the features of each time step in the second visual feature sequence are used as the query vector. The cosine similarity between the query vector and each key memory vector is calculated. The cosine similarity is then scaled, exponentially operated, and normalized over all memory slots to obtain the addressing weight of each memory slot. The memory vectors of each value are weighted and summed according to the addressing weights to obtain the memory audio feature sequence used to compensate for the visual modality; and the memory audio feature sequence is added to the second visual feature sequence to obtain the enhanced visual feature sequence.

[0009] Furthermore, the multi-head key-value memory network includes multiple memory heads, each memory head having several key memory vectors and corresponding value memory vectors, wherein: Each time step feature in the second visual feature sequence is written into the key memory vector of each memory head, and each time step feature in the second audio feature sequence is written into the corresponding value memory vector, so as to save the lip shape feature in the key memory unit and the corresponding audio feature in the value memory unit; During the query phase, for any time step feature in the second visual feature sequence, in each memory head, the query vector is obtained through linear transformation, and the cosine similarity between the query vector and each key memory vector in the corresponding memory head is calculated; each cosine similarity is multiplied by a preset scaling factor and then exponentially calculated, and the results corresponding to all key memory vectors are normalized to obtain the addressing weight of the corresponding memory head. For each memory head, the value memory vectors within the memory head are weighted and summed according to the corresponding addressing weights to obtain the compensated audio features of the corresponding memory head; the compensated audio features of all memory heads are concatenated along the feature dimension and subjected to linear transformation to obtain the memory audio feature sequence; the memory audio feature sequence is added to the second visual feature sequence and subjected to layer normalization to obtain the enhanced visual feature sequence.

[0010] Furthermore, the enhanced visual feature sequence and the second audio feature sequence are input into the joint cross-modal fusion module for fusion, specifically including: The enhanced visual feature sequence is concatenated with the second audio feature sequence along the feature dimension to obtain a joint audiovisual feature sequence; A first correlation matrix is ​​obtained by weighting the enhanced visual feature sequence and the joint audiovisual feature sequence and then performing a nonlinear transformation; a second correlation matrix is ​​obtained by weighting the second audio feature sequence and the joint audiovisual feature sequence and then performing a nonlinear transformation. The attention weights of the visual modality are calculated based on the first correlation matrix and the enhanced visual feature sequence, and the enhanced visual feature sequence is weighted and connected by residuals to obtain a weighted visual feature sequence; the attention weights of the audio modality are calculated based on the second correlation matrix and the second audio feature sequence, and the second audio feature sequence is weighted and connected by residuals to obtain a weighted audio feature sequence. The weighted visual feature sequence and the weighted audio feature sequence are concatenated along the feature dimension to obtain the fused feature sequence.

[0011] Furthermore, the calculation of the first correlation matrix, the second correlation matrix, and the corresponding attention weights respectively includes: The enhanced visual feature sequence is transposed and multiplied by the first trainable weight matrix, and then multiplied by the joint audiovisual feature sequence. The result is scaled by the square root of the feature dimension and then input into the hyperbolic tangent nonlinear function to obtain the first correlation matrix. The first correlation matrix is ​​used to characterize the degree of correlation within the visual modality and between the visual modality and the joint audiovisual modality. The second audio feature sequence is transposed and multiplied by the second trainable weight matrix, and then multiplied by the joint audiovisual feature sequence. The result is scaled by the square root of the feature dimension and then input into the hyperbolic tangent nonlinear function to obtain the second correlation matrix. The second correlation matrix is ​​used to characterize the degree of correlation within the audio modality and between the audio modality and the joint audiovisual modality. Using the first correlation matrix and the enhanced visual feature sequence, the attention weight of the visual modality is calculated using a preset linear transformation weight matrix, and the enhanced visual feature sequence is weighted and residual connected using the attention weight to obtain a weighted visual feature sequence. Using the second correlation matrix and the second audio feature sequence, the attention weight of the audio modality is calculated using a preset linear transformation weight matrix, and the second audio feature sequence is weighted and residually connected using the attention weight to obtain the weighted audio feature sequence.

[0012] Furthermore, the encoder network, the sequence-to-sequence decoding branch, and the connection-time classification decoding branch specifically include: The encoder network includes several encoder layers based on the self-attention mechanism. Each encoder layer is configured with a multi-head self-attention sub-layer and a feedforward network sub-layer in sequence. Each sub-layer adopts a residual connection and layer normalization structure. Each sub-layer is used to perform global temporal modeling on the fused feature sequence to obtain the encoded feature sequence. The sequence-to-sequence decoding branch includes several decoder layers based on the self-attention mechanism. Each decoder layer is equipped with a self-attention sub-layer, an attention sub-layer between the encoder and the decoder, and a feedforward network sub-layer. The sequence-to-sequence decoding branch uses the generated prefix in the target character sequence as input to predict the current character, and calculates the first decoding loss based on the cross-entropy between the predicted character distribution and the target character. The connection-time classification decoding branch includes multiple one-dimensional convolutional layers with non-linear activation functions and layer normalization operations. These layers are used to smooth the encoded feature sequence in the time dimension and output the character category probability distribution at each time step, and calculate the connection-time classification loss based on the probability distribution.

[0013] Furthermore, the audio front-end network, the audio back-end network, the visual front-end network, the visual back-end network, the multi-head key-value memory network, the joint cross-modal fusion module, the encoder network, and the decoding branch constitute an audiovisual speech recognition model. The parameters of the audiovisual speech recognition model are obtained through training, which includes: Using the weighted sum of memory reconstruction loss, memory contrast loss, and hybrid connection temporal classification / attention loss based on the first and second character probability sequences as the total loss function, the parameters of the audio front-end network, the audio back-end network, the visual front-end network, the visual back-end network, the multi-head key-value memory network, the joint cross-modal fusion module, the encoder network, and the decoding branch are optimized through backpropagation to obtain the audiovisual speech recognition model that is robust in noisy environments. The audiovisual speech recognition model is then used to perform recognition on noisy speech signals and their corresponding lip video sequences, and the recognized text results are output. The hybrid connection-time classification / attention loss is obtained by weighting and summing the connection-time classification loss and the first decoding loss according to a preset weight coefficient. The first decoding loss is calculated by the sequence-to-sequence decoding branch, and the connection-time classification loss is calculated by the connection-time classification decoding branch.

[0014] Furthermore, when performing recognition on noisy speech signals and their corresponding lip video sequences, the process also includes: In the sequence-to-sequence decoding branch, the log probabilities of multiple candidate output character sequences are recursively calculated based on the encoded feature sequence. Simultaneously, in the connection-time classification decoding branch, the log probabilities of multiple candidate output character sequences are calculated based on the same encoded feature sequence. The two log probabilities are weighted and summed according to a preset decoding weight coefficient. The weighted log probabilities are used as a scoring function to perform a beam search on a set of candidate output character sequences with a preset beam width. The candidate output character sequence with the highest score is selected as the recognized text result.

[0015] According to a second aspect of this disclosure, an artificial intelligence-based speech translation system is provided, the system comprising: The preprocessing module is used to acquire audiovisual samples containing the speech signal of the target speaker and a synchronized lip video sequence, and to preprocess the audiovisual samples to obtain a normalized speech waveform and a corresponding lip image sequence. The audio feature extraction module is used to input the normalized speech waveform into the audio front-end network, perform temporal convolution and self-supervised feature extraction to obtain a first audio feature sequence; input the first audio feature sequence into the audio back-end network, reduce the dimensionality through one-dimensional convolution and perform temporal modeling to obtain a second audio feature sequence for speech content representation; The visual feature extraction module is used to input the lip image sequence into the visual front-end network, and obtain a first visual feature sequence through three-dimensional convolution and spatial feature extraction; the first visual feature sequence is input into the visual back-end network, and a second visual feature sequence is obtained through one-dimensional convolution dimensionality reduction. The compensation module is used to establish a one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence using a multi-head key-value memory network to obtain an enhanced visual feature sequence; The feature fusion module is used to input the enhanced visual feature sequence and the second audio feature sequence into the joint cross-modal fusion module for fusion to obtain a fused feature sequence; The text recognition module is used to input the fused feature sequence into the encoder network for temporal encoding to obtain the encoded feature sequence; input the encoded feature sequence into the sequence decoding branch and the connection temporal classification decoding branch respectively; and use the first character probability sequence output by the sequence to sequence decoding branch and the second character probability sequence output by the connection temporal classification decoding branch to generate a text sequence corresponding to the audiovisual sample. The translation module is used to perform sentence segmentation, punctuation restoration, and text normalization on the text sequence to obtain a source language text sequence. The source language text sequence is then input into a neural machine translation model for semantic mapping to generate a target language text sequence corresponding to the source language text sequence, and the target language text sequence is output.

[0016] The technical solution disclosed herein has the following beneficial effects: By constructing an audiovisual joint framework for speech translation, the speech signal of the target speaker is combined with synchronous lip-sync video. Temporal features are extracted from both audio and visual modalities. A one-to-many mapping relationship between visual and audio features is established using a multi-head key-value memory network to compensate and enhance visual information. Fine-grained alignment and collaborative modeling are achieved through joint cross-modal fusion, improving the accuracy and robustness of text sequences in noisy scenarios from the source. Furthermore, the generated text sequences are processed by sentence segmentation, punctuation restoration, and text normalization to form a clearly structured source language text sequence. This source language text sequence is then input into a neural machine translation model for semantic mapping, outputting a target language text sequence. This reduces the transmission and amplification of recognition errors in the cascaded system to the translation results, improving the semantic consistency and readability of the translation results. It is suitable for applications requiring real-time subtitles and cross-language understanding, such as classroom teaching and meeting recording. Attached Figure Description

[0017] Figure 1 This is a flowchart of an artificial intelligence-based speech translation method as described in the embodiments of this specification; Figure 2 This is a structural block diagram of an artificial intelligence-based speech translation system as described in the embodiments of this specification. Detailed Implementation

[0018] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0019] Furthermore, the accompanying drawings are merely illustrative of this disclosure. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0020] This invention provides a speech translation method for products based on artificial intelligence. (See reference...) Figure 1 The diagram shown illustrates a flowchart of an artificial intelligence-based speech translation method according to an embodiment of the present invention. This method can be applied to devices such as personal computers, servers, tablets, and mobile phones. The method can be executed by a device, which can be implemented by software and / or hardware. Specifically, the method may include the following steps S101-S107: In step S101, an audiovisual sample containing the target speaker's speech signal and a synchronized lip video sequence is obtained. The audiovisual sample is preprocessed to obtain a normalized speech waveform and a corresponding lip image sequence.

[0021] The process of acquiring the lip image sequence includes: performing face detection and facial key point detection on the audiovisual samples to locate the speaker's mouth region; cropping a fixed-size image block centered on the mouth region as the lip image sequence; converting the lip image sequence into a grayscale image; and normalizing it based on the overall mean and variance of all the lip image sequences.

[0022] Specifically, when acquiring the speech signal, the speech signal is sampled at a sampling rate of 16kHz, and the normalized speech waveform is divided into continuous audio data frames with a frame length of 20 milliseconds.

[0023] In this embodiment, step S101 first acquires an audiovisual sample containing the target speaker's speech signal and a synchronized lip video sequence. Specifically, this can be achieved by synchronously acquiring data using a camera and microphone, such as when a teacher or student speaks into a tablet during an online class, obtaining the original video frame sequence and the corresponding original audio waveform data. To ensure consistency in subsequent processing, the audio waveform uses a mono audio signal with an original sampling rate of 16kHz, and the video frame rate remains constant so that the video frames and audio frames can be aligned one-to-one on the time axis. For the video portion, to accurately extract the target speaker's lip region, face detection and facial key point detection are performed on each frame. In a preferred implementation, facial key points in each frame can be detected and tracked, and the approximate position of the mouth can be determined through these key points. Considering that the speaker may have slight head rotation and forward / backward displacement in scenarios such as classrooms, in order to reduce the impact of face rotation and scale changes, the key point trajectory can be interpolated and smoothed in the time dimension. For example, an interpolation and frame smoothing method with a window width of 12 frames can be used to compensate for some frames that are unstable or temporarily missed, thereby obtaining a smoother and more continuous mouth movement trajectory.

[0024] After achieving stable localization of the speaker's mouth region, a fixed-size image block is cropped centered on this mouth region to form a lip image sequence. In one specific implementation, a 120×120 pixel rectangular bounding box can be used to crop the region of interest (ROI) of the mouth on the original video frame, ensuring that the bounding box always covers the speaker's lip region and a small number of related areas around it throughout the speaking process, forming a temporally sequential sequence of local lip images. To reduce the redundancy of color information and highlight the shape and movement contours of the lips, all cropped lip image frames can be converted to grayscale images. Furthermore, to reduce the impact of differences in imaging conditions such as brightness and contrast between different samples on feature extraction, each grayscale image frame can be normalized based on the overall mean and variance of all lip image frames in the training set. This ensures that the normalized lip image sequence maintains a stable numerical distribution, making it more suitable as input for subsequent visual feature extraction networks. In this embodiment, the lip image sequence is temporally synchronized with the original video, and its frame number corresponds one-to-one with the corresponding audio timeline, providing a foundation for the alignment and joint modeling of audiovisual features.

[0025] For the speech signal, in step S101, the original speech waveform in the audiovisual sample is preprocessed. Specifically, the speech signal of the target speaker is first sampled at a sampling rate of 16kHz to obtain a continuous-time digitized speech waveform. To eliminate the differences in overall energy and DC bias between different recording segments, each original audio waveform is normalized to meet the statistical characteristics of zero mean and unit variance, thus obtaining a normalized speech waveform. Based on this, the normalized speech waveform is divided into continuous audio data frames according to a preset frame length. In a preferred setting, each frame contains 20 milliseconds of audio data, that is, the speech signal is divided into frames on the time axis using a sliding window of fixed duration, resulting in a multi-frame temporally arranged audio data frame sequence. Through the above processing, on the one hand, the resolution of the audio modality in the time dimension is guaranteed, which facilitates the subsequent network to model speech features at the frame level; on the other hand, it also ensures that each frame of audio data is synchronously matched with the lip image of the corresponding time period in time, thus providing a unified and standardized input data format for subsequent audiovisual joint recognition.

[0026] In step S102, the normalized speech waveform is input into the audio front-end network, and temporal convolution and self-supervised feature extraction are performed to obtain the first audio feature sequence; the first audio feature sequence is input into the audio back-end network, and dimensionality reduction and temporal modeling are performed through one-dimensional convolution to obtain the second audio feature sequence used for speech content representation.

[0027] In this embodiment, the audio front-end network is used to perform temporal convolution and self-supervised feature extraction on the normalized speech waveform obtained in step S101. Specifically, the aforementioned normalized speech waveform is directly input into a self-supervised pre-trained speech feature extraction model in chronological order. This model is based on a one-dimensional convolutional layer and a multi-layer encoder structure. The one-dimensional convolutional layer first downsamples and models the continuous speech waveform locally in time, mapping the original waveform to a high-dimensional latent speech representation space, resulting in an intermediate feature sequence arranged in time steps. Subsequently, stacked encoder layers (e.g., a self-attention-based transformer encoder layer) perform contextual modeling and feature enhancement on these intermediate features over a long period of time, capturing the pronunciation patterns and semantic relevance across multiple audio frames in the speech signal, thereby outputting a first audio feature sequence that changes over time. To facilitate alignment with subsequent visual modalities, in one exemplary implementation, the structure and parameters of the audio front-end network can be set to be consistent with existing self-supervised speech representation models (such as the wav2vec 2.0 model pre-trained on large-scale unlabeled speech data). That is, its one-dimensional convolutional feature extraction module and stacked transformer encoder are transferred as a whole to the audio front-end network, so that the audio feature vector output at each time step has a dimension of 1024, and the number of time steps is aligned with the video time step Tv after front-end processing, thereby forming a first audio feature sequence with a length of Tv and 1024 dimensions at each time step on the time axis.

[0028] After obtaining the first audio feature sequence, this sequence is input into the audio backend network. Dimensionality reduction and temporal modeling are performed using one-dimensional convolution to obtain the second audio feature sequence for speech content representation. Specifically, the audio backend network first includes at least one one-dimensional convolutional layer to linearly transform and compress the first audio feature sequence in the feature dimension, mapping the feature dimension of each time step from 1024 to a preset model dimension of 512. This unifies the scale of the audio and visual modalities in the feature dimension, resulting in an intermediate audio feature sequence with a length of Tv and 512 dimensions at each time step. Based on this, the audio backend network further employs a self-attention-based encoder structure as the main body of acoustic modeling, performing global temporal modeling of the intermediate audio feature sequence through a multi-layer transformer encoder. Each transformer encoder layer includes a multi-head self-attention sublayer and a feedforward network sublayer, combined with residual connections and layer normalization operations. This allows the model to simultaneously focus on the correlations at different time positions in the speech sequence, thereby adaptively aggregating different phonemes, syllables, and word-level information in the temporal dimension. After processing by a multi-layer encoder, the output feature sequence at each time step not only contains local acoustic information but also encodes the contextual relationship of the entire speech. This output serves as the second audio feature sequence for speech content representation, providing a high-level audio representation foundation for subsequent association modeling and decoding recognition with visual modalities.

[0029] In step S103, the lip image sequence is input into the visual front-end network, and a first visual feature sequence is obtained through three-dimensional convolution and spatial feature extraction; the first visual feature sequence is input into the visual back-end network, and a second visual feature sequence is obtained through one-dimensional convolution dimensionality reduction.

[0030] When acquiring the second visual feature sequence, the process includes: in the visual front-end network, performing spatiotemporal convolution on the lip image sequence through a three-dimensional convolutional layer to extract local spatiotemporal features, and extracting deep spatial features through a convolutional neural network with residual blocks as basic units to obtain the first visual feature sequence; in the visual back-end network, compressing the feature dimension of the first visual feature sequence through a one-dimensional convolutional layer to obtain the second visual feature sequence.

[0031] In this embodiment, step S103 uses the lip image sequence obtained in step S101 as input to the visual modality, feeding it into the visual front-end network for 3D convolution and spatial feature extraction to obtain the first visual feature sequence. The role of the visual front-end network is to simultaneously characterize the motion changes of the lips in the temporal dimension and the shape differences in the spatial dimension, so that the output representation can reflect the mouth shape changes corresponding to different pronunciations. In an exemplary implementation, the visual front-end network adopts a cascaded structure of "3D convolutional layer + deep convolutional neural network", that is, firstly, a 3D convolutional layer is used to perform spatiotemporal convolution on the lip image sequence. This 3D convolutional layer slides the convolution kernel simultaneously in the time, horizontal and vertical dimensions, thereby extracting the spatiotemporal local features of the lip region within a local time window. To fully utilize existing large-scale visual representations, this embodiment uses a self-supervised visual feature extraction model, MoCo v2, as the backbone of the subsequent deep convolutional neural network. The original first two-dimensional convolutional layer is truncated and replaced with the aforementioned three-dimensional convolutional layer, ensuring that the output tensor of the three-dimensional convolutional layer maintains consistency with the input of the first residual block (ResBlock) of MoCo v2 in terms of channel count and spatial size. In this way, the three-dimensional convolutional layer is responsible for extracting local dynamic patterns in both temporal and spatial dimensions, while the subsequent stacked residual blocks in MoCo v2 further extract deeper visual features in the spatial dimension, resulting in a first visual feature sequence containing rich lip texture and shape information at each time step. In terms of feature dimension settings, the input lip image sequence can be represented as a tensor of size Tv×112×112×1 (Tv is the time step of the lip sequence). After passing through a three-dimensional convolutional layer, an intermediate representation of size Tv×28×28×64 is obtained. After processing by the MoCo v2 deep network, the spatial dimension is compressed through operations such as global pooling to form a first visual feature sequence of length Tv and 2048 dimensions at each time step.

[0032] After obtaining the first visual feature sequence, it is input into the visual backend network. The feature dimension is compressed using one-dimensional convolution to obtain the second visual feature sequence for subsequent audiovisual modeling. The visual backend network employs a one-dimensional convolutional layer along the time dimension, mapping the 2048-dimensional visual features at each time step to the target model dimension through a convolutional kernel. In a typical configuration, the feature dimension is compressed to 512 dimensions, achieving a linear transformation from Tv×2048 to Tv×512 using a one-dimensional convolutional layer. Since one-dimensional convolution shares kernel parameters in the time direction, it not only reduces the feature dimension but also smooths and integrates local correlations between adjacent time steps. This allows the output second visual feature sequence to retain key lip information while removing some redundant and noise components. The resulting second visual feature sequence corresponds one-to-one with the original lip image sequence in terms of time steps and maintains the same feature dimension as the audio feature dimension output by the audio backend network in step S102, facilitating alignment, association, and fusion with the audio modality in subsequent steps.

[0033] In step S104, a one-to-many mapping relationship is established between the second visual feature sequence and the second audio feature sequence using a multi-head key-value memory network to obtain an enhanced visual feature sequence. Specifically, this includes: writing the second visual feature sequence into the key memory unit of the multi-head key-value memory network, and writing the second audio feature sequence into the corresponding value memory unit; during querying, using the features at each time step in the second visual feature sequence as query vectors, calculating the cosine similarity between the query vector and each key memory vector, and performing scaling, exponential operation, and normalization on all memory slots to obtain the addressing weights of each memory slot; performing a weighted summation of each value memory vector according to the addressing weights to obtain a memory audio feature sequence used to compensate for the visual modality; and adding the memory audio feature sequence to the second visual feature sequence to obtain the enhanced visual feature sequence.

[0034] The multi-head key-value memory network comprises multiple memory heads, each memory head having several key memory vectors and corresponding value memory vectors. Specifically: features at each time step of the second visual feature sequence are written into the key memory vectors of each memory head, and features at each time step of the second audio feature sequence are written into the corresponding value memory vectors, so as to store lip-shape features in the key memory units and corresponding audio features in the value memory units; during the query phase, for any time step feature in the second visual feature sequence, in each memory head, the query vector is obtained through a linear transformation, and the query vector is compared with the corresponding memory head's... The cosine similarity between each key memory vector is calculated; each cosine similarity is multiplied by a preset scaling factor and then exponentially calculated, and the results corresponding to all key memory vectors are normalized to obtain the addressing weights of the corresponding memory heads; for each memory head, the value memory vectors within the memory head are weighted and summed according to the corresponding addressing weights to obtain the compensated audio features of the corresponding memory head; the compensated audio features of all memory heads are concatenated along the feature dimension and linearly transformed to obtain the memory audio feature sequence; the memory audio feature sequence is added to the second visual feature sequence and layer normalization is performed to obtain the enhanced visual feature sequence.

[0035] In this embodiment, step S104 utilizes a multi-head key-value memory network to establish a one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence. This is used to "recall" the rich audio representation corresponding to the current lip shape from the audio modality and compensate for the visual modality to obtain an enhanced visual feature sequence. Specifically, the second visual feature sequence obtained in steps S102 and S103 is denoted as... The second audio feature sequence is denoted as They are aligned frame-by-frame in the time dimension. Multi-head key-value memory networks consist of two parts: key memory and value memory, where the key memory... It is a multi-head structure, with each memory head corresponding to a set of key memory vectors, which can be represented as follows: Value memory Where N is the number of memory slots and D is the model dimension. This refers to the subspace dimension of each memory head. During the actual writing process, the visual features from each time step in the second visual feature sequence are... The audio features are fed into a key memory network to store lip-shape related features in key memory units; the audio features at each time step in the second audio feature sequence are then processed. The data is fed into a value memory network to store the audio features corresponding to these mouth shapes in the value memory units. Through this paired storage, a correspondence is established between visual features and audio features at the same time location in the key and value memory spaces, thereby characterizing a one-to-many mapping of a mouth shape to multiple possible pronunciations.

[0036] During the query phase, each time step feature in the current second visual feature sequence to be enhanced is used as the query vector. A multi-head key-value memory network is then used to retrieve the audio feature most relevant to the lip shape from the value memory. For the visual feature at time step j... In each memory head, a linear transformation is first performed to obtain the query vector corresponding to that head. Then calculate the query vector and the first one in the memory header. Key memory vector The cosine similarity between the two is used to obtain a similarity-based addressing score. The addressing score is calculated using similarity-based addressing, and its corresponding addressing weight... It can be represented as: ; in, The cosine similarity metric function is used. Let h be the projection weight matrix corresponding to the h-th memory head. The scaling factor is used. Using the above formula, the normalized addressing weight distribution of the visual feature at time step j relative to all memory slots can be obtained in each memory head, i.e., assigning greater weights to which key memory slots are more similar. Then, using these addressing weights as coefficients, a weighted sum is performed on the value memory vectors in the corresponding value memory, extracting a compensated audio feature from each memory head. That is, each memory head extracts a possible audio compensation representation from the value memory to characterize the diversity of different speech pronunciations under the same mouth shape.

[0037] To integrate the compensated audio features output from each memory head into a unified model space, this embodiment uses a set of embedding weights to aggregate the multi-head outputs. Specifically, the compensated audio features corresponding to all memory heads are concatenated along their feature dimensions to obtain a vector. Then through the embedding weight matrix Perform a linear transformation to obtain memory audio features consistent with the model dimensions. The calculation relationship can be expressed as: ; in, This represents the compensated audio features extracted from the value memory using the h-th memory head. To aggregate the embedding weight matrices from different head outputs, This is a sequence composed of the memorized audio features from all time steps. For each time step, the corresponding memorized audio features are... With original secondary visual features The results are added together to obtain visual features with added audio compensation information. Then, the result is subjected to layer normalization to form an enhanced visual feature sequence. Through the above processing of the multi-head key-value memory network, the visual modality, while maintaining its own mouth shape representation, introduces semantically related audio feature information. Thus, in subsequent audiovisual fusion and recognition in noisy environments, the discriminative information of the audio modality can be more fully utilized to compensate for the ambiguity that may exist in pure lip reading features.

[0038] In step S105, the enhanced visual feature sequence and the second audio feature sequence are input into the joint cross-modal fusion module for fusion to obtain a fused feature sequence, specifically including: The enhanced visual feature sequence is concatenated with the second audio feature sequence along the feature dimension to obtain a joint audiovisual feature sequence; wherein, the enhanced visual feature sequence obtained in the aforementioned steps is denoted as... The second audio feature sequence is Where B represents the batch size (or the flattened length of the time steps after batch processing), and D represents the feature dimension (e.g., 512-dimensional). The enhanced visual feature sequence and the second audio feature sequence are concatenated along the feature dimension to obtain the joint audiovisual feature sequence. Its form is: ; in This represents a cascading operation along the feature dimension. This concatenation operation allows for the simultaneous preservation of information that enhances both visual and audio features within the same feature space, providing a foundation for subsequent calculations of the correlation between the two modalities.

[0039] A first correlation matrix is ​​obtained by weighting the enhanced visual feature sequence and the joint audiovisual feature sequence and performing a nonlinear transformation; a second correlation matrix is ​​obtained by weighting the second audio feature sequence and the joint audiovisual feature sequence and performing a nonlinear transformation. The first correlation matrix is ​​obtained by transposing the enhanced visual feature sequence, multiplying it by a first trainable weight matrix, and then multiplying it by the joint audiovisual feature sequence. The result is scaled by the square root of the feature dimension and then input into a hyperbolic tangent nonlinear function. The first correlation matrix is ​​used to characterize the correlation within the visual modality and between the visual modality and the joint audiovisual modality. The second correlation matrix is ​​obtained by transposing the second audio feature sequence, multiplying it by a second trainable weight matrix, and then multiplying it by the joint audiovisual feature sequence. The result is scaled by the square root of the feature dimension and then input into a hyperbolic tangent nonlinear function. The second correlation matrix is ​​used to characterize the correlation within the audio modality and between the audio modality and the joint audiovisual modality.

[0040] In this process, after obtaining the joint audiovisual feature sequence, a first correlation matrix and a second correlation matrix are introduced to simultaneously characterize the correlation between the visual modality and the joint audiovisual modality, as well as the correlation between the audio modality and the joint audiovisual modality. Specifically, enhanced visual feature sequences are utilized. With joint audiovisual feature sequences By weighted product and nonlinear transformation, a joint correlation matrix of visual modalities is constructed. Its definition is: ; in, This is the first trainable weight matrix, used to establish a learnable mapping between visual features and joint audiovisual features. For a hyperbolic tangent nonlinear function, the denominator contains... This corresponds to scaling along the feature dimension, serving to stabilize the numerical values ​​and normalize the scale. Similarly, the second audio feature sequence is utilized. With joint audiovisual feature sequences The weighted relationships are used to construct the joint correlation matrix of audio modes. Its definition is: ; in, This is the second trainable weight matrix, used to characterize the correlation between the audio modality and the joint audiovisual modality. The first correlation matrix is ​​obtained through the above two equations. Second correlation matrix It not only characterizes the semantic correlation within each visual and audio modality, but also reflects the interaction between them and the joint audiovisual modal features. The higher the correlation coefficient, the stronger the consistency of the corresponding samples across modalities.

[0041] The attention weights of the visual modality are calculated based on the first correlation matrix and the enhanced visual feature sequence, and a weighted sum and residual connection are performed on the enhanced visual feature sequence to obtain a weighted visual feature sequence; the attention weights of the audio modality are calculated based on the second correlation matrix and the second audio feature sequence, and a weighted sum and residual connection are performed on the second audio feature sequence to obtain a weighted audio feature sequence; wherein, the calculation includes: Using the first correlation matrix and the enhanced visual feature sequence, the attention weight of the visual modality is calculated using a preset linear transformation weight matrix, and the enhanced visual feature sequence is weighted and residual connected using the attention weight to obtain a weighted visual feature sequence. Using the second correlation matrix and the second audio feature sequence, the attention weight of the audio modality is calculated using a preset linear transformation weight matrix, and the second audio feature sequence is weighted and residually connected using the attention weight to obtain the weighted audio feature sequence.

[0042] After obtaining the correlation matrices for both modalities, attention weights for the visual and audio modalities are further calculated based on these matrices to highlight feature dimensions that are more important in cross-modal interaction. For the visual modality, the first correlation matrix is ​​used. and enhanced visual feature sequences Attention weights for the visual modality are calculated using a trainable weight matrix. The formula for its calculation is: ; in, The weight matrix is ​​a linearly transformed matrix with a preset dimension and updated through training. To modify the activation function of the linear unit, An attention weight map representing the visual modality is used to indicate which visual features need to be strengthened or suppressed at different time points. Similarly, for the audio modality, a second correlation matrix is ​​used. Second audio feature sequence It employs a trainable linear transformation weight matrix. Calculate the attention weights for the audio modality Its form is: ; in, This is the trainable weight matrix corresponding to the audio modality. This represents an attention map of audio modalities. By introducing a correlation matrix into the calculation of attention weights, the attention distribution can not only depend on the features of a single modality itself, but also explicitly consider the correlation with another modality and joint modalities, thereby more effectively mining cross-modal complementary information.

[0043] After obtaining the visual modality attention weights Zv and audio modality attention weights Za, these attention weights are used to perform a weighted summation and residual connection operation on the features of the two modalities, resulting in a weighted visual feature sequence and a weighted audio feature sequence. Specifically, the weighted features of the visual modality... We obtain it from the following formula: ; Weighted features of audio modalities We obtain it from the following formula: ; in, This is a trainable linear transformation matrix used to map attention weights back to the feature space. The plus sign corresponds to the residual connection operation, which superimposes the original modality features onto the new features after attention modulation to preserve the original information and avoid gradient vanishing during training. After this step, This is a weighted visual feature sequence, corresponding to the weighted visual feature sequence in the claims. This is a weighted audio feature sequence, corresponding to the weighted audio feature sequence.

[0044] The weighted visual feature sequence and the weighted audio feature sequence are concatenated along the feature dimension to obtain the fused feature sequence. Finally, the weighted visual modal features and audio modal features are concatenated along the feature dimension to obtain the final fused feature sequence. Its form is: ; This fused feature sequence contains both attention-weighted visual and audio features, and explicitly utilizes the correlation information between the two modalities and the joint audiovisual modality during the construction process. This allows the visual modality to play a more significant supporting role for the audio modality in noisy environments such as classrooms, providing a more discriminative and robust joint audiovisual representation for subsequent encoding and decoding recognition.

[0045] In step S106, the fused feature sequence is input into the encoder network for temporal encoding to obtain the encoded feature sequence; the encoded feature sequence is input into the sequence decoding branch and the connection temporal classification decoding branch respectively, and the first character probability sequence output by the sequence to sequence decoding branch and the second character probability sequence output by the connection temporal classification decoding branch are used to generate the text sequence corresponding to the audiovisual sample.

[0046] The encoder network, the sequence-to-sequence decoding branch, and the connection-time classification decoding branch specifically include: the encoder network comprising several encoder layers based on a self-attention mechanism, each encoder layer having a multi-head self-attention sub-layer and a feedforward network sub-layer sequentially, and each sub-layer employing residual connections and layer normalization structures, each sub-layer being used to perform global temporal modeling on the fused feature sequence to obtain the encoded feature sequence; the sequence-to-sequence decoding branch comprising several decoder layers based on a self-attention mechanism, each decoder layer having a self-attention sub-layer, an attention sub-layer between the encoder and decoder, and a feedforward network sub-layer, the sequence-to-sequence decoding branch using the generated prefix in the target character sequence as input to predict the current character, and calculating the first decoding loss based on the cross-entropy between the predicted character distribution and the target character; the connection-time classification decoding branch comprising multiple one-dimensional convolutional layers with non-linear activation functions and layer normalization operations, which are used to smooth the encoded feature sequence in the temporal dimension and output the character category probability distribution at each time step, and calculate the connection-time classification loss based on the probability distribution.

[0047] In this embodiment, the fused feature sequence obtained in step S106 is first used as the input to the encoder network. Since the fused features output by the aforementioned joint cross-modal fusion module are 1024-dimensional, while the underlying temporal modeling network uses a 512-dimensional model, a linear transformation layer is used to map the fused features at each time step from 1024-dimensional to 512-dimensional before entering the encoder network, ensuring consistency with the dimensions of the subsequent network structure. The fused feature sequence after the linear transformation retains the original time step length, changing only in the number of feature channels. Subsequently, this feature sequence is sequentially input into several encoder layers based on a self-attention mechanism. Each layer includes a multi-head self-attention sublayer and a feedforward network sublayer, with residual connections and layer normalization operations introduced outside each sublayer to model the entire audiovisual feature over a global time range. Through the multi-layer stacked encoder network, long-term dependencies and global context information can be effectively captured, resulting in the encoded feature sequence. , where T is the time step length of the fused feature sequence, and the encoded features will be used by both decoding branches simultaneously in the future.

[0048] In the sequence-to-sequence decoding branch, a Transformer decoder structure based on a self-attention mechanism is adopted, specifically comprising six stacked decoder layers. Each decoder layer includes a self-attention sub-layer for the generated target character sequence, an encoder-decoder attention sub-layer that aggregates the encoded feature sequence, and a feedforward network sub-layer, also used in conjunction with residual connections and layer normalization. During the training phase, a character-level teacherforcing strategy can be employed. After adding a start marker to the beginning of the target character sequence, it is fed into the decoder sequentially. At each step, the generated prefix character is used as input, combined with the encoded feature sequence, and the encoded features at different time positions are weighted and converged through the attention mechanism to output the character probability distribution of the current step, achieving autoregressive modeling. Let the input frame sequence be... The target character sequence is Where L is the target character length, the posterior probability given by the autoregressive decoder can be expressed as: ; in This represents the generated character prefixes preceding the current position. Based on this posterior probability, the sequence-to-sequence decoding branch minimizes the negative log-likelihood of the target character sequence through cross-entropy loss, thus obtaining the first decoding loss, corresponding to the first character probability sequence and its training target.

[0049] In the Connected Temporal Classification (CTC) decoding branch, to directly output the character category probability distribution at each time step on the time axis, several one-dimensional convolutional layers are further superimposed on the output of the encoded feature sequence. Non-linear activation functions (ReLU) and layer normalization structures are set between each convolutional layer to smooth the encoded features in the time dimension and enhance local temporal modeling capabilities. After multiple convolutions and normalization processes, linear mapping and Softmax activation are used to convert the hidden representation at each time step into a probability distribution defined on the character label set (including letters, numbers, whitespace, etc.), thus forming the second character probability sequence. Using the Connected Temporal Classification (CTC) approach, assuming that the output conditions at each time step are independent, the posterior probability of the path given the input sequence x is approximately: ; in Let represent the label output at time step t (including blank labels). Based on this, the connection-temporal classification decoding branch obtains the connection-temporal classification loss by summing all possible alignment paths and minimizing the corresponding negative log-likelihood; this is the training objective of the connection-temporal classification decoding branch. Since this branch directly constrains frame-by-frame prediction on the time axis, it can effectively supervise the encoded features even when alignment information is unclear.

[0050] In reasoning and generating text sequences, this embodiment simultaneously utilizes the first character probability sequence output by the sequence-to-sequence decoding branch and the second character probability sequence output by the temporal classification decoding branch to model the character sequence corresponding to the same audiovisual sample. The sequence-to-sequence decoding branch focuses on the overall language structure and contextual dependencies through an autoregressive approach, making it better at capturing long-distance semantic constraints; the temporal classification decoding branch directly provides stable frame-level alignment information and peak position constraints in the time dimension. By combining the probability information obtained from the two decoding branches, the candidate character sequences are comprehensively scored and selected, which can alleviate the bias caused by a single decoding method, improve the recognition robustness in noisy environments such as classrooms, and thus generate the final text sequence corresponding to the input audiovisual sample.

[0051] In step S107, the text sequence is processed by sentence segmentation, punctuation restoration and text normalization to obtain a source language text sequence. The source language text sequence is then input into a neural machine translation model for semantic mapping to generate a target language text sequence corresponding to the source language text sequence, and the target language text sequence is output.

[0052] During translation, the text sequence is first segmented into sentences to form sentence-level units that better meet the requirements of machine translation input. Sentence segmentation can employ at least one of the following strategies: Sentence segmentation based on time alignment information: The pause point is inferred by using the time alignment / peak position (e.g., CTC path peak or alignment boundary) generated by the decoding branch in step S106. When the pause duration exceeds the threshold or the confidence changes abruptly, the sentence boundary is inserted. Rule-based sentence segmentation: When the length of consecutive characters exceeds a preset limit (e.g., 40-80 Chinese characters), it is segmented by combining function words / modal words (such as "then", "because", "therefore") or line breaks; Model-based sentence segmentation: Train a lightweight sentence segmentation model (such as Transformer / BiLSTM) to predict whether each position is a sentence boundary and output several sentence segments.

[0053] After sentence segmentation, punctuation restoration is performed on the resulting sentence-level sequence to obtain punctuated text. A punctuation restoration model can be used to predict the punctuation category (such as "comma, period, question mark, exclamation mark, pause mark, colon, semicolon, no punctuation") for each character (or word) position and insert the predicted punctuation into the corresponding position. The input to the punctuation restoration model may include: the segmented text sequence, and auxiliary information including: confidence level, pause / duration features, etc., from step S106, to improve the punctuation accuracy of interrogative sentences, compound sentences, and other structures. The output is then a sequence of sentences with punctuation, for example, restoring "Today we talk about the development trend of artificial intelligence" to "Today, we talk about the development trend of artificial intelligence".

[0054] After adding punctuation marks, the sentence sequence with punctuation is then normalized to obtain the source language text sequence. Normalization processes may include, but are not limited to: Standardization of numbers and units: Convert colloquial numbers (such as "one hundred and twenty-three" and "three point five") into standard numerical forms ("123" and "3.5"), and unify the expression of units ("kilograms" and "meters"). Time / Date Standardization: Standardize "Tomorrow at 3 PM" and "December 2025" into a unified format (e.g., "2025-12-26 15:00" and "2025-12"). Proper noun and abbreviation handling: Correct the capitalization or spacing of English abbreviations and proper nouns (e.g., “chatgpt” → “ChatGPT”); Noise word cleanup: Remove filler words (such as "um," "ah," "that") or demote their ranking; Character set and symbol unification: full-width / half-width characters, Chinese / English punctuation unification, simplified / traditional character conversion (if needed), and compression of redundant spaces.

[0055] The result is a source language text sequence with a clear structure, complete punctuation, and uniform format (which can be a list organized by sentences or a spliced ​​paragraph text).

[0056] In the specific translation process, the source language text sequence is input into a neural machine translation (NMT) model for semantic mapping to generate a target language text sequence. Specifically, this may include: Perform word segmentation / sub-word encoding (such as BPE / SentencePiece) on the source language text sequence and construct the model input; An encoder-decoder architecture (such as Transformer) is used to model the context of the source language and generate a sequence of target language tokens at the decoding end; During inference, beamsearch or other decoding strategies are used to obtain the target language sequence with the highest probability; The target language token sequence is de-encoded into natural language text to form a target language text sequence; Output the target language text sequence (e.g., display it on a terminal, write it to a subtitle file, or use it as input for a subsequent speech synthesis module).

[0057] In one embodiment, the audio front-end network, the audio back-end network, the visual front-end network, the visual back-end network, the multi-head key-value memory network, the joint cross-modal fusion module, the encoder network, and the decoding branch constitute an audiovisual speech recognition model. The parameters of the audiovisual speech recognition model are obtained through training, which includes: Using the weighted sum of memory reconstruction loss, memory contrast loss, and hybrid connection temporal classification / attention loss based on the first and second character probability sequences as the total loss function, the parameters of the audio front-end network, the audio back-end network, the visual front-end network, the visual back-end network, the multi-head key-value memory network, the joint cross-modal fusion module, the encoder network, and the decoding branch are optimized through backpropagation to obtain the audiovisual speech recognition model that is robust in noisy environments. The audiovisual speech recognition model is then used to perform recognition on noisy speech signals and their corresponding lip video sequences, and the recognized text results are output. The hybrid connection-time classification / attention loss is obtained by weighting and summing the connection-time classification loss and the first decoding loss according to a preset weight coefficient. The first decoding loss is calculated by the sequence-to-sequence decoding branch, and the connection-time classification loss is calculated by the connection-time classification decoding branch.

[0058] The audio front-end network, audio back-end network, visual front-end network, visual back-end network, multi-head key-value memory network, joint cross-modal fusion module, encoder network, and decoding branch together constitute the audiovisual speech recognition model. All parameters of this model are automatically learned through the training process. During the training phase, for the same batch of audiovisual samples, the second audio feature sequence and the second visual feature sequence are first extracted according to the aforementioned steps. The fused feature sequence is obtained through the multi-head key-value memory network and the joint cross-modal fusion module, and the first character probability sequence and the second character probability sequence are output through the encoder network and two decoding branches, respectively. Based on this, this embodiment constructs a total loss composed of multiple loss functions, and performs joint optimization of the entire set of network parameters through backpropagation, so that the model can both store discriminative audio representations in the memory module and achieve stable and accurate audiovisual recognition at the character level.

[0059] To ensure that value memory units in a multi-head key-value memory network retain correct and representative audio representations, this embodiment introduces memory reconstruction loss and memory contrast loss. The memory reconstruction loss guarantees that when audio features are used as queries, the value memory can reconstruct a representation close to the original audio features; it is defined based on cosine similarity as follows: ; in, This represents the raw audio features from the audio backend network. This represents the audio features reconstructed through value memory. Let be the audio vector of the i-th memory slot in the memory. This refers to the addressing weights obtained when using audio features as queries. Let be the cosine similarity function. Minimize ... This allows value memory to restore the input audio features as much as possible when obtaining correct addressing, thus ensuring that the memory cell stores a representation consistent with the original audio. Memory contrast loss, on the other hand, is used to constrain the audio representations between different memory slots to distinguish them and reduce redundancy in memory content; it is defined as: ; in The value memory vectors are located in different memory slots. By encouraging a reduction in cosine similarity between slots, each memory slot tends to store different audio patterns, thereby improving the discriminative power of the entire value memory space. The combined effect of these two losses enables the multi-head key-value memory network to reliably save and recall diverse and representative audio features after training, for use in compensating for visual modalities.

[0060] In the character-level recognition part, this embodiment employs a hybrid connection-based temporal classification / attention loss, corresponding to the joint training of the connection-based temporal classification decoding branch and the sequence-to-sequence decoding branch. Let the input frame sequence be... The target character sequence is , where T and L are the input and target lengths, respectively. The connection-time classification part, assuming independent output conditions at each time step, approximates the posterior probability of the path as the product of the probabilities at each time step: ; Based on this probability definition, the connection-based temporal classification loss is used. The autoregressive decoding part directly estimates the posterior probability of the sequence using the chain rule, and is trained using cross-entropy loss. Its posterior probability is expressed as: ; in This represents the character prefix preceding the current position. Combining the memory reconstruction loss, memory contrast loss, and the two recognition losses mentioned above, this embodiment constructs the total loss function: ; in The preset weights are used to balance the connection-time classification loss and the sequence-to-sequence attention loss. During training, with this total loss function as the optimization objective, backpropagation is used to jointly update the parameters of the audio front-end network, audio back-end network, visual front-end network, visual back-end network, multi-head key-value memory network, joint cross-modal fusion module, encoder network, and decoding branch, thereby obtaining an audiovisual speech recognition model with strong robustness in noisy environments, especially classroom background noise.

[0061] As a supplement, in the above embodiments, when performing recognition on the noisy speech signal and its corresponding lip video sequence, the method further includes: In the sequence-to-sequence decoding branch, the log probabilities of multiple candidate output character sequences are recursively calculated based on the encoded feature sequence. Simultaneously, in the connection-time classification decoding branch, the log probabilities of multiple candidate output character sequences are calculated based on the same encoded feature sequence. The two log probabilities are weighted and summed according to a preset decoding weight coefficient. The weighted log probabilities are used as a scoring function to perform a beam search on a set of candidate output character sequences with a preset beam width. The candidate output character sequence with the highest score is selected as the recognized text result.

[0062] During the inference phase, for a new noisy speech signal and its corresponding lip-sync video sequence, this embodiment first uses a trained audiovisual speech recognition model to extract a second audio feature sequence and a second visual feature sequence according to a predetermined process. An enhanced visual feature sequence is then obtained through a multi-head key-value memory network. Finally, a second character probability sequence based on connection-time classification and a first character probability sequence based on sequence-to-sequence decoding are obtained through a joint cross-modal fusion module, an encoder network, and two decoding branches. To fully utilize the complementary advantages of the two decoding methods, a combined CTC / attention one-time bundle search strategy is adopted during decoding. The set of candidate output character sequences is denoted as... The final optimal character sequence is then selected. satisfy: ; in, These are weighting coefficients used in the decoding stage to balance the log probabilities of the two decoding branches. Specifically, given a beam width, the sequence-to-sequence decoding branch recursively generates the log probabilities of multiple candidate output character sequences based on the encoded feature sequences. Simultaneously, the temporal classification decoding branch calculates the corresponding CTC log probabilities for these candidate sequences based on the same encoded feature sequences. Then, the two log probabilities are weighted and summed according to preset decoding weighting coefficients. The weighted score is used as the scoring function in the beam search, repeatedly expanding and filtering the candidate set, ultimately selecting the candidate output character sequence with the highest score as the recognized text result. This joint decoding method can fully utilize linguistic context information while ensuring temporal alignment stability, enabling reliable audiovisual recognition text output even in complex environments such as classroom noise.

[0063] Based on the same line of thought, such as Figure 2 The diagram shown is a structural block diagram of an artificial intelligence-based speech translation system according to an embodiment of the present invention. The system includes: The preprocessing module 201 is used to acquire audiovisual samples containing the speech signal of the target speaker and a synchronized lip video sequence, and to preprocess the audiovisual samples to obtain a normalized speech waveform and a corresponding lip image sequence. The audio feature extraction module 202 is used to input the normalized speech waveform into the audio front-end network, perform temporal convolution and self-supervised feature extraction to obtain a first audio feature sequence; input the first audio feature sequence into the audio back-end network, reduce the dimensionality through one-dimensional convolution and perform temporal modeling to obtain a second audio feature sequence for speech content representation. The visual feature extraction module 203 is used to input the lip image sequence into the visual front-end network, and obtain a first visual feature sequence through three-dimensional convolution and spatial feature extraction; and input the first visual feature sequence into the visual back-end network, and obtain a second visual feature sequence through one-dimensional convolution dimensionality reduction. Compensation module 204 is used to establish a one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence using a multi-head key-value memory network to obtain an enhanced visual feature sequence; Feature fusion module 205 is used to input the enhanced visual feature sequence and the second audio feature sequence into the joint cross-modal fusion module for fusion to obtain a fused feature sequence; The text recognition module 206 is used to input the fused feature sequence into the encoder network for temporal encoding to obtain the encoded feature sequence; input the encoded feature sequence into the sequence decoding branch and the connection temporal classification decoding branch respectively; and use the first character probability sequence output by the sequence to sequence decoding branch and the second character probability sequence output by the connection temporal classification decoding branch to generate a text sequence corresponding to the audiovisual sample. The translation module 207 is used to perform sentence segmentation, punctuation restoration and text normalization on the text sequence to obtain a source language text sequence, input the source language text sequence into a neural machine translation model for semantic mapping, generate a target language text sequence corresponding to the source language text sequence, and output the target language text sequence.

[0064] This system constructs an audiovisual joint framework for speech translation, combining the target speaker's speech signal with synchronized lip-sync video. Temporal features are extracted from both audio and visual modalities. A multi-head key-value memory network is used to establish a one-to-many mapping relationship between visual and audio features, compensating for and enhancing visual information. Fine-grained alignment and collaborative modeling are achieved through joint cross-modal fusion, improving the accuracy and robustness of text sequences in noisy environments from the source. Furthermore, the generated text sequences undergo sentence segmentation, punctuation restoration, and text normalization to form a clearly structured source language text sequence. This source language text sequence is then input into a neural machine translation model for semantic mapping, outputting a target language text sequence. This reduces the propagation and amplification of recognition errors in the cascaded system to the translation results, improving the semantic consistency and readability of the translation. It is suitable for applications requiring real-time subtitles and cross-language understanding, such as classroom teaching and meeting recording.

[0065] The specific details of the above system have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0066] The accompanying drawings are merely illustrative of the processes included in the methods according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the drawings do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0067] It should be noted that although several modules or units of the system have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0068] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0069] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A speech translation method based on artificial intelligence, characterized in that, The method includes: Acquire audiovisual samples containing the target speaker's speech signal and a synchronized lip video sequence; preprocess the audiovisual samples to obtain a normalized speech waveform and a corresponding lip image sequence. The normalized speech waveform is input into the audio front-end network, where temporal convolution and self-supervised feature extraction are performed to obtain the first audio feature sequence. The first audio feature sequence is then input into the audio back-end network, where dimensionality reduction is achieved through one-dimensional convolution and temporal modeling to obtain the second audio feature sequence used for speech content representation. The lip image sequence is input into the visual front-end network, and a first visual feature sequence is obtained through three-dimensional convolution and spatial feature extraction; the first visual feature sequence is input into the visual back-end network, and a second visual feature sequence is obtained through one-dimensional convolution dimensionality reduction. A one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence is established using a multi-head key-value memory network to obtain an enhanced visual feature sequence; The enhanced visual feature sequence and the second audio feature sequence are input into the joint cross-modal fusion module for fusion to obtain the fused feature sequence; The fused feature sequence is input into the encoder network for temporal encoding to obtain the encoded feature sequence. The encoded feature sequence is then input into the sequence decoding branch and the connection temporal classification decoding branch, respectively. The first character probability sequence output by the sequence-to-sequence decoding branch and the second character probability sequence output by the connection temporal classification decoding branch are used to generate the text sequence corresponding to the audiovisual sample. The text sequence is segmented, punctuation restored, and text normalized to obtain a source language text sequence. The source language text sequence is then input into a neural machine translation model for semantic mapping to generate a target language text sequence corresponding to the source language text sequence, and the target language text sequence is output.

2. The artificial intelligence-based speech translation method according to claim 1, characterized in that, When acquiring the lip image sequence, the following steps are included: By performing face detection and facial key point detection on the audiovisual samples, the speaker's mouth region is located, and a fixed-size image block is cropped with the mouth region as the center as the lip image sequence. The lip image sequence is converted into a grayscale image and normalized according to the overall mean and variance of all the lip image sequences. When acquiring the second visual feature sequence, the following steps are included: In the visual front-end network, the lip image sequence is spatiotemporally convolved by a three-dimensional convolutional layer to extract local spatiotemporal features, and deep spatial features are extracted by a convolutional neural network with residual blocks as basic units to obtain the first visual feature sequence. In the visual backend network, the feature dimension of the first visual feature sequence is compressed by a one-dimensional convolutional layer to obtain the second visual feature sequence.

3. The artificial intelligence-based speech translation method according to claim 1, characterized in that, When establishing a one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence, the following steps are included: The second visual feature sequence is written into the key memory unit of the multi-head key-value memory network, and the second audio feature sequence is written into the corresponding value memory unit. During the query, the features of each time step in the second visual feature sequence are used as the query vector. The cosine similarity between the query vector and each key memory vector is calculated. The cosine similarity is then scaled, exponentially operated, and normalized over all memory slots to obtain the addressing weight of each memory slot. The memory vectors of each value are weighted and summed according to the addressing weights to obtain the memory audio feature sequence used to compensate for the visual modality; and the memory audio feature sequence is added to the second visual feature sequence to obtain the enhanced visual feature sequence.

4. The artificial intelligence-based speech translation method according to claim 3, characterized in that, The multi-head key-value memory network includes multiple memory heads, each memory head having several key memory vectors and corresponding value memory vectors, wherein: Each time step feature in the second visual feature sequence is written into the key memory vector of each memory head, and each time step feature in the second audio feature sequence is written into the corresponding value memory vector, so as to save the lip shape feature in the key memory unit and the corresponding audio feature in the value memory unit; During the query phase, for any time step feature in the second visual feature sequence, in each memory head, the query vector is obtained through linear transformation, and the cosine similarity between the query vector and each key memory vector in the corresponding memory head is calculated; each cosine similarity is multiplied by a preset scaling factor and then exponentially calculated, and the results corresponding to all key memory vectors are normalized to obtain the addressing weight of the corresponding memory head. For each memory head, the value memory vectors within the memory head are weighted and summed according to the corresponding addressing weights to obtain the compensated audio features of the corresponding memory head; the compensated audio features of all memory heads are concatenated along the feature dimension and subjected to linear transformation to obtain the memory audio feature sequence; the memory audio feature sequence is added to the second visual feature sequence and subjected to layer normalization to obtain the enhanced visual feature sequence.

5. The artificial intelligence-based speech translation method according to claim 1, characterized in that, The enhanced visual feature sequence and the second audio feature sequence are input into the joint cross-modal fusion module for fusion, specifically including: The enhanced visual feature sequence is concatenated with the second audio feature sequence along the feature dimension to obtain a joint audiovisual feature sequence; A first correlation matrix is ​​obtained by weighting the enhanced visual feature sequence and the joint audiovisual feature sequence and then performing a nonlinear transformation; a second correlation matrix is ​​obtained by weighting the second audio feature sequence and the joint audiovisual feature sequence and then performing a nonlinear transformation. The attention weights of the visual modality are calculated based on the first correlation matrix and the enhanced visual feature sequence, and the enhanced visual feature sequence is weighted and connected by residuals to obtain a weighted visual feature sequence; the attention weights of the audio modality are calculated based on the second correlation matrix and the second audio feature sequence, and the second audio feature sequence is weighted and connected by residuals to obtain a weighted audio feature sequence. The weighted visual feature sequence and the weighted audio feature sequence are concatenated along the feature dimension to obtain the fused feature sequence.

6. The artificial intelligence-based speech translation method according to claim 5, characterized in that, The calculation of the first correlation matrix, the second correlation matrix, and the corresponding attention weights includes: The enhanced visual feature sequence is transposed and multiplied by the first trainable weight matrix, and then multiplied by the joint audiovisual feature sequence. The result is scaled by the square root of the feature dimension and then input into the hyperbolic tangent nonlinear function to obtain the first correlation matrix. The first correlation matrix is ​​used to characterize the degree of correlation within the visual modality and between the visual modality and the joint audiovisual modality. The second audio feature sequence is transposed and multiplied by the second trainable weight matrix, and then multiplied by the joint audiovisual feature sequence. The result is scaled by the square root of the feature dimension and then input into the hyperbolic tangent nonlinear function to obtain the second correlation matrix. The second correlation matrix is ​​used to characterize the degree of correlation within the audio modality and between the audio modality and the joint audiovisual modality. Using the first correlation matrix and the enhanced visual feature sequence, the attention weight of the visual modality is calculated using a preset linear transformation weight matrix, and the enhanced visual feature sequence is weighted and residual connected using the attention weight to obtain a weighted visual feature sequence. Using the second correlation matrix and the second audio feature sequence, the attention weight of the audio modality is calculated using a preset linear transformation weight matrix, and the second audio feature sequence is weighted and residually connected using the attention weight to obtain the weighted audio feature sequence.

7. The artificial intelligence-based speech translation method according to claim 1, characterized in that, The encoder network, the sequence-to-sequence decoding branch, and the connection-time classification decoding branch specifically include: The encoder network includes several encoder layers based on the self-attention mechanism. Each encoder layer is configured with a multi-head self-attention sub-layer and a feedforward network sub-layer in sequence. Each sub-layer adopts a residual connection and layer normalization structure. Each sub-layer is used to perform global temporal modeling on the fused feature sequence to obtain the encoded feature sequence. The sequence-to-sequence decoding branch includes several decoder layers based on the self-attention mechanism. Each decoder layer is equipped with a self-attention sub-layer, an attention sub-layer between the encoder and the decoder, and a feedforward network sub-layer. The sequence-to-sequence decoding branch uses the generated prefix in the target character sequence as input to predict the current character, and calculates the first decoding loss based on the cross-entropy between the predicted character distribution and the target character. The connection-time classification decoding branch includes multiple one-dimensional convolutional layers with non-linear activation functions and layer normalization operations. These layers are used to smooth the encoded feature sequence in the time dimension and output the character category probability distribution at each time step, and calculate the connection-time classification loss based on the probability distribution.

8. The artificial intelligence-based speech translation method according to claim 1, characterized in that, The audio front-end network, the audio back-end network, the visual front-end network, the visual back-end network, the multi-head key-value memory network, the joint cross-modal fusion module, the encoder network, and the decoding branch constitute the audiovisual speech recognition model. The parameters of the audiovisual speech recognition model are obtained through training, which includes: Using the weighted sum of memory reconstruction loss, memory contrast loss, and hybrid connection temporal classification / attention loss based on the first and second character probability sequences as the total loss function, the parameters of the audio front-end network, the audio back-end network, the visual front-end network, the visual back-end network, the multi-head key-value memory network, the joint cross-modal fusion module, the encoder network, and the decoding branch are optimized through backpropagation to obtain the audiovisual speech recognition model that is robust in noisy environments. The audiovisual speech recognition model is then used to perform recognition on noisy speech signals and their corresponding lip video sequences, and the recognized text results are output. The hybrid connection-time classification / attention loss is obtained by weighting and summing the connection-time classification loss and the first decoding loss according to a preset weight coefficient. The first decoding loss is calculated by the sequence-to-sequence decoding branch, and the connection-time classification loss is calculated by the connection-time classification decoding branch.

9. The artificial intelligence-based speech translation method according to claim 8, characterized in that, When performing recognition on noisy speech signals and their corresponding lip video sequences, the following is also included: In the sequence-to-sequence decoding branch, the log probabilities of multiple candidate output character sequences are recursively calculated based on the encoded feature sequence. Simultaneously, in the connection-time classification decoding branch, the log probabilities of multiple candidate output character sequences are calculated based on the same encoded feature sequence. The two log probabilities are weighted and summed according to a preset decoding weight coefficient. The weighted log probabilities are used as a scoring function to perform a beam search on a set of candidate output character sequences with a preset beam width. The candidate output character sequence with the highest score is selected as the recognized text result.

10. An artificial intelligence-based speech translation system, the system comprising: The preprocessing module is used to acquire audiovisual samples containing the speech signal of the target speaker and a synchronized lip video sequence, and to preprocess the audiovisual samples to obtain a normalized speech waveform and a corresponding lip image sequence. The audio feature extraction module is used to input the normalized speech waveform into the audio front-end network, perform temporal convolution and self-supervised feature extraction to obtain a first audio feature sequence; input the first audio feature sequence into the audio back-end network, reduce the dimensionality through one-dimensional convolution and perform temporal modeling to obtain a second audio feature sequence for speech content representation; The visual feature extraction module is used to input the lip image sequence into the visual front-end network, and obtain a first visual feature sequence through three-dimensional convolution and spatial feature extraction; the first visual feature sequence is input into the visual back-end network, and a second visual feature sequence is obtained through one-dimensional convolution dimensionality reduction. The compensation module is used to establish a one-to-many mapping relationship between the second visual feature sequence and the second audio feature sequence using a multi-head key-value memory network to obtain an enhanced visual feature sequence; The feature fusion module is used to input the enhanced visual feature sequence and the second audio feature sequence into the joint cross-modal fusion module for fusion to obtain a fused feature sequence; The text recognition module is used to input the fused feature sequence into the encoder network for temporal encoding to obtain the encoded feature sequence; The encoded feature sequence is input into the sequence decoding branch and the connection temporal classification decoding branch respectively. The first character probability sequence output by the sequence to sequence decoding branch and the second character probability sequence output by the connection temporal classification decoding branch are used to generate the text sequence corresponding to the audiovisual sample. The translation module is used to perform sentence segmentation, punctuation restoration, and text normalization on the text sequence to obtain a source language text sequence. The source language text sequence is then input into a neural machine translation model for semantic mapping to generate a target language text sequence corresponding to the source language text sequence, and the target language text sequence is output.

Images