Audio and video layered coding translation method and system for low-bandwidth cross-border scenarios

Abstract

The application relates to the fields of audio-video communication and cross-language translation technology, and discloses an audio-video layered coding translation method and system for a low-bandwidth cross-border scene, which has the technical scheme as follows: original audio data and original video frame data are acquired, preprocessing data packets are obtained through content analysis preprocessing, a layered code stream set is obtained through independent scalable layered coding based on the preprocessing data packets, a sending data packet set is obtained through priority layered transmission scheduling based on the layered code stream set and network feedback information, and a video frame sequence is obtained through layered decoding, time alignment and cross-modal fusion translation based on the sending data packet set. The application realizes audio-visual double-channel information guarantee for a translation task in a low-bandwidth cross-border scene through priority layered transmission scheduling and cross-modal fusion translation based on preprocessing data packets and a layered code stream set, and improves translation robustness.

Description

Technical Field

[0001] This invention relates to the fields of audio and video communication and cross-language translation technology, specifically to an audio and video layered coding translation method and system for low-bandwidth cross-border scenarios. Background Technology

[0002] With the rapid development of cross-language real-time communication scenarios such as cross-border remote conferencing and cross-border online education, real-time audio and video translation technology has become an important foundation for supporting barrier-free cross-border communication. Affected by factors such as long cross-border transmission links and complex routing nodes, cross-border communication generally suffers from bandwidth limitations, drastic latency fluctuations, and high packet loss rates, placing stringent demands on the stability and robustness of audio and video translation services. Existing technologies generally adopt a scheme combining scalable audio and video coding with layered transmission. The basic layer ensures minimum communication quality, while the enhancement layer adapts to scenarios with ample bandwidth as needed. Some schemes also introduce region-of-interest (ROI) coding strategies to prioritize the clarity of facial areas. However, existing optimization systems are mostly designed around the human viewing experience, and the core translation task still heavily relies on acoustic single-modal input. The visual-audio information contained in the video is not included in the redundancy guarantee mechanism of the translation link. When network deterioration leads to audio signal damage, the quality of speech recognition and translation will significantly decline, and service continuity cannot be guaranteed in extreme scenarios. Therefore, existing technologies have shortcomings. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide an audio and video layered coding translation method and system for low-bandwidth cross-border scenarios. This method involves obtaining preprocessed data packets through content analysis and preprocessing, generating a layered bitstream set based on independent and scalable layered coding, executing priority layered transmission scheduling based on network feedback to obtain a set of transmitted data packets, and then obtaining a video frame sequence through layered decoding, time alignment, and cross-modal fusion translation. This achieves a multimodal layered adaptive transmission architecture for translation tasks, solving the problems of over-reliance on single-modal audio translation and drastic degradation of translation quality under network fluctuations in low-bandwidth, high-packet-loss cross-border scenarios. It also improves the availability of translation services and the overall system resilience under extreme bandwidth constraints.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a layered encoding translation method for audio and video in low-bandwidth cross-border scenarios, comprising:

[0006] The raw audio data and raw video frame data are acquired, and preprocessed data packets are obtained through content analysis and preprocessing.

[0007] Based on the preprocessed data packets, a set of layered bitstreams is obtained through independent scalable layered coding.

[0008] Based on the hierarchical bitstream set and network feedback information, the set of data packets to be sent is obtained through priority-based hierarchical transmission scheduling.

[0009] Based on the set of transmitted data packets, a video frame sequence is obtained through layered decoding, time alignment, and cross-modal fusion translation.

[0010] Furthermore, the raw audio data and raw video frame data are acquired, and preprocessed data packets are obtained through content analysis and preprocessing, including:

[0011] Acquire raw audio data and raw video frame data;

[0012] Based on the original audio data and original video frame data, parallel video analysis and audio detection are used to obtain the lip grayscale image sequence, facial bounding box coordinates and speech frame marker sequence.

[0013] Based on the original audio data, original video frame data, lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence, a preprocessed data packet is obtained through timestamp alignment and data encapsulation.

[0014] Furthermore, the process of obtaining a lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence through parallel video analysis and audio detection based on the original audio data and original video frame data includes:

[0015] Based on the original video frame data, a lightweight face detection network and a lip feature point regression model are used to obtain the lip grayscale image sequence and facial bounding box coordinates.

[0016] Based on the original audio data, a speech frame marker sequence containing speech frames and silence frames is obtained through a speech activity detection algorithm.

[0017] Furthermore, the step of obtaining a layered bitstream set based on the preprocessed data packet through independent scalable layered coding includes:

[0018] Based on the original audio data and original video frame data, an audio layered bitstream containing the basic audio layer bitstream and the audio enhancement layer bitstream is obtained by disabling the inter-frame dependency audio encoder for layered encoding.

[0019] Based on the original video frame data and facial bounding box coordinates, a scalable video coding method with reduced quantization parameters for the facial region is used to obtain a video layered bitstream containing a basic video layer bitstream and a video enhancement layer bitstream.

[0020] Based on the grayscale image sequence of the lips, a visual speech layer data packet sequence containing the basic visual speech layer code stream and the visual speech enhancement layer code stream is obtained by using a visual speech feature extraction network and a two-level residual vector quantization hierarchical coding.

[0021] Based on the audio layered bitstream, video layered bitstream, and visual-speech layer data packet sequences, a set of layered bitstreams is obtained by organizing and encapsulating them according to type.

[0022] Furthermore, based on the lip grayscale image sequence, a visual-speech layer data packet sequence is obtained by using a visual-speech feature extraction network and two-level residual vector quantization hierarchical coding, which includes a visual-speech base layer bitstream and a visual-speech enhancement layer bitstream, comprising:

[0023] Based on the grayscale image sequence of the lips, a visual speech feature vector sequence is obtained through a visual speech feature extraction network cascaded with a three-dimensional spatiotemporal convolutional network and a bidirectional long short-term memory network.

[0024] Based on the visual speech feature vector sequence, coarse quantization is performed through first-level residual vector quantization to obtain the visual speech basic layer code stream and the first-level quantization residual vector sequence;

[0025] Based on the first-level quantized residual vector sequence, the visual-speech enhancement layer bitstream is obtained by fine quantization through the second-level residual vector quantization.

[0026] Based on the basic visual-speech layer bitstream and the visual-speech enhancement layer bitstream, the visual-speech layer data packet sequence is obtained by aligning the timestamps to the range of covered audio frames.

[0027] Furthermore, the step of obtaining the set of data packets to be sent through priority-based hierarchical transmission scheduling based on the hierarchical bitstream set and network feedback information includes:

[0028] Based on network feedback information, a congestion control algorithm that jointly estimates the delay gradient and packet loss rate is used to obtain an estimate of the available bandwidth.

[0029] Based on the hierarchical bitstream set, priority sending tags are obtained through translation task priority classification rules;

[0030] Based on priority sending tags and available bandwidth estimates, a set of data packets is obtained through priority-based hierarchical transmission scheduling.

[0031] Furthermore, the process of obtaining a video frame sequence based on the transmitted data packet set through layered decoding, time alignment, and cross-modal fusion translation includes:

[0032] Based on the set of transmitted data packets, the decoded audio PCM signal, decoded video frame sequence, and visual-speech feature sequence are obtained through layered decoding and time alignment.

[0033] Based on decoded audio PCM signals and visual-speech feature sequences, target language translated text is obtained through acoustic feature extraction, robust audio-visual fusion recognition, and neural machine translation.

[0034] Based on the decoded video frame sequence, visual and speech feature sequence, and target language translated text, a lip-smoothing interpolation is performed on the missing frames using a lip-shape generation model to obtain the video frame sequence.

[0035] Furthermore, the step of obtaining the decoded audio PCM signal, decoded video frame sequence, and visual-speech feature sequence based on the transmitted data packet set through layered decoding and time alignment includes:

[0036] Based on the set of transmitted data packets, the decoded audio PCM signal is obtained through the Opus standard decoder;

[0037] Based on the set of transmitted data packets, the sequence of decoded video frames is obtained through the H.264 scalable video decoder.

[0038] Based on the set of transmitted data packets, the decoded visual-speech feature vector sequence is obtained through a residual vector quantization decoder.

[0039] Based on the decoded visual-speech feature vector sequence and its audio frame time range information, the feature frame rate is extended to match the audio frame rate through linear interpolation to obtain the visual-speech feature sequence.

[0040] Furthermore, the process of obtaining the target language translated text based on decoded audio PCM signals and visual-speech feature sequences through acoustic feature extraction, robust audiovisual fusion recognition, and neural machine translation includes:

[0041] Based on the decoded audio PCM signal, acoustic feature sequences are obtained through Mel filter bank analysis.

[0042] Based on acoustic feature sequences and visual-speech feature sequences, an audio-visual fusion recognition model with a dual encoder plus fusion decoder architecture and a modality dropout strategy introduced during training is used to obtain the source language recognition text sequence.

[0043] Based on the source language, the text sequence is identified, and the target language translation text is obtained through neural machine translation.

[0044] Secondly, the present invention provides an audio and video layered coding translation system for low-bandwidth cross-border scenarios, comprising:

[0045] Data preprocessing module: used to acquire raw audio data and raw video frame data, and obtain preprocessed data packets through content analysis and preprocessing;

[0046] Layered encoding module: Based on the preprocessed data packet, a set of layered bitstreams is obtained through independent scalable layered encoding;

[0047] Transmission scheduling module: Based on the hierarchical bitstream set and network feedback information, it obtains the set of data packets to be sent through priority-based hierarchical transmission scheduling;

[0048] Fusion Translation Module: Based on the set of sent data packets, video frame sequences are obtained through layered decoding, time alignment, and cross-modal fusion translation.

[0049] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0050] This invention is based on audio and video content analysis and preprocessing and independent scalable layered coding. It dynamically adapts to changes in cross-border link bandwidth through priority layered transmission scheduling, and completes cross-modal fusion translation based on layered decoding and time alignment results. It realizes an adaptive audio and video translation transmission system with multimodal redundancy capabilities, which solves the problems of over-reliance on acoustic single-modality translation in low-bandwidth cross-border scenarios and sharp decline in service quality when the network deteriorates. It improves the continuous availability of translation functions and the system's resistance to fluctuations under extreme network conditions, and enhances the robustness of the translation system and the continuity of cross-language communication. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the steps of the audio and video layered coding translation method for low-bandwidth cross-border scenarios of the present invention.

[0052] Figure 2 Flowchart of steps to obtain preprocessed data packets;

[0053] Figure 3 Flowchart of steps to obtain a layered bitstream set;

[0054] Figure 4 A flowchart outlining the steps to obtain the set of data packets to be sent.

[0055] Figure 5 Flowchart of steps to obtain a video frame sequence;

[0056] Figure 6 This is a schematic diagram of the structure of the audio and video layered coding translation system for low-bandwidth cross-border scenarios according to the present invention. Detailed Implementation

[0057] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof.

[0058] Identical parts are indicated by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, while the terms "bottom surface," "top surface," "inner," and "outer" refer to directions toward or away from the geometric center of a specific part, respectively.

[0059] The term "and / or" in the following text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0060] Example 1:

[0061] like Figure 1 As shown, this embodiment provides a layered audio and video coding translation method for low-bandwidth cross-border scenarios, including:

[0062] The raw audio data and raw video frame data are acquired, and preprocessed data packets are obtained through content analysis and preprocessing.

[0063] Based on preprocessed data packets, a set of layered bitstreams is obtained through independent scalable layered coding.

[0064] Based on the hierarchical bitstream set and network feedback information, the set of data packets to be sent is obtained through priority-based hierarchical transmission scheduling.

[0065] Based on the set of transmitted data packets, a video frame sequence is obtained through layered decoding, time alignment, and cross-modal fusion translation.

[0066] The original audio data consists of continuous audio samples in pulse code modulation (PCM) format, acquired synchronously via a microphone through the operating system's standard acquisition interface and a network time protocol clock. The sampling rate is typically 16kHz, with each processing frame lasting 20ms, and each frame carrying a timestamp aligned with a unified time series. The original video frame data is a continuously acquired sequence of digital image frames, acquired synchronously via a camera through the operating system's standard acquisition interface and a network time protocol clock. The frame rate is typically 25fps, with each processing frame lasting 40ms, and each frame carrying a unified timestamp aligned with the original audio data. Content analysis preprocessing is a lightweight analysis process performed in parallel on the audio and video data. It is used to extract key visual and speech region information and speech activity markers required for the translation task. The system comprises two parallel processing steps: lip region extraction and speech activity detection. The preprocessed data package is a set of data packages obtained after unified timestamp alignment and structured encapsulation, including raw audio data, raw video frame data, lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence. The lip grayscale image sequence is a fixed-resolution lip image sequence obtained after face detection, lip keypoint localization, outward cropping, and grayscale processing, with a single frame resolution typically set to 64×64 pixels. The facial bounding box coordinates are the rectangular coordinates of the speaker's facial region output by a lightweight face detection network, used for region of interest quantization adjustment in video coding. The speech frame marker sequence is a binary marker sequence of speech frames and silence frames for each audio frame obtained through speech activity detection, used to guide the bitrate allocation for audio coding.

[0067] Independent Scalable Layered Coding is a three-layer parallel coding mechanism that adds independent visual and speech scalable bitstreams to traditional audio and video scalable coding. It generates multi-layered bitstreams that can be independently decoded and prioritized for discarding to adapt to cross-border link bandwidth fluctuations. This includes layered coding using an audio encoder based on the Opus encoder, scalable video coding based on the H.264 / SVC standard, and two-level residual vector quantization layered coding based on visual and speech features. The Opus encoder is an open-source audio encoder for low-latency real-time speech scenarios, supporting layered coding modes. H.264 / SVC is a scalable video coding extension standard based on the H.264 standard, supporting temporal, spatial, and quality scalability. Two-level residual vector quantization layered coding is an encoding method that performs coarse and fine quantization on the visual and speech feature vector sequences. The first level generates a basic feature bitstream that can be independently decoded, and the second level generates an enhanced residual bitstream to improve feature accuracy. The layered bitstream set is the complete code obtained by organizing and encapsulating the three types of layered bitstreams according to their types. The stream set specifically includes the audio base layer stream, audio enhancement layer stream, video base layer stream, video enhancement layer stream, visual-speech base layer stream, and visual-speech enhancement layer stream. Network feedback information consists of cross-border link status data periodically transmitted back by the receiver via a real-time transmission control protocol, including arrival timestamps and packet loss rates periodically fed back by the receiver, used by the sender to estimate the available bandwidth of the current link in real time. Priority-based hierarchical transmission scheduling is a bandwidth adaptive scheduling mechanism centered on the minimum availability of the translation task. It is used to filter the streams to be sent according to a preset priority order based on the currently available bandwidth to ensure the basic input for the translation function under extremely low bandwidth conditions. The preset priority order from high to low is: audio base layer, visual-speech base layer, video base layer, audio enhancement layer, visual-speech enhancement layer, and video enhancement layer. The data packet set to be sent is the set of data packets to be sent after priority scheduling filtering, whose total data volume does not exceed the currently available bandwidth. Under any bandwidth conditions, audio base layer and visual-speech base layer data packets are preferentially retained.

[0068] Layered decoding refers to the independent decoding processing performed by the receiving end on each layer of the incoming bitstream, used to recover the audio signal, video image, and visual-speech feature vectors of the corresponding layer. Specifically, it includes Opus standard audio decoding, H.264 / SVC standard video decoding, and residual vector quantization visual-speech feature decoding. Time alignment is a timing calibration process performed on decoded data of different frame rates and modalities based on the timestamp information carried in the data packets. It is used to adjust the visual-speech features and audio features to a frame-by-frame aligned state, providing a unified timing benchmark for cross-modal fusion. Cross-modal fusion translation is a process of fusing and recognizing acoustic features and visual-speech features and converting them into target language translation text. It is used to maintain a stable translation accuracy even when low bandwidth causes a decline in single-modal quality. Specifically, it includes three stages: acoustic feature extraction, robust audio-visual fusion automatic speech recognition, and neural machine translation. The video frame sequence is a continuous video image sequence obtained after decoding reconstruction, missing frame lip interpolation completion, and lip-sync processing, with lip movement effects synchronized with the translation result and decoded audio.

[0069] This embodiment performs content analysis preprocessing on the raw audio and video data and extracts visual and speech features independent of video encoding. It constructs an independent encoding architecture that includes three types of scalable layered bitstreams: audio, video, and visual and speech. It adapts to cross-border low-bandwidth fluctuating links through a priority-based layered transmission scheduling mechanism centered on the translation task. Through layered decoding and cross-modal audiovisual fusion translation, it achieves stable translation output with multimodal redundancy. This ensures the minimum availability of audio and video translation services in cross-border low-bandwidth scenarios, improves the accuracy of translation recognition and the overall system's resilience under extreme bandwidth constraints and network packet loss conditions, and also takes into account the visual continuity of video images and the synchronized presentation of lip movements.

[0070] Furthermore, this embodiment provides a step of acquiring raw audio data and raw video frame data, and obtaining a preprocessed data packet through content analysis preprocessing, including:

[0071] Acquire raw audio data and raw video frame data;

[0072] Based on the original audio data and original video frame data, parallel video analysis and audio detection are used to obtain lip grayscale image sequences, facial bounding box coordinates, and speech frame marker sequences; specifically including:

[0073] Based on the original video frame data, a lightweight face detection network and a lip feature point regression model are used to obtain the lip grayscale image sequence and facial bounding box coordinates; based on the original audio data, a speech frame tag sequence containing speech frames and silence frames is obtained through a speech activity detection algorithm.

[0074] Based on the original audio data, original video frame data, lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence, a preprocessed data packet is obtained through timestamp alignment and data encapsulation.

[0075] Among them, parallel video analysis and audio detection are two types of lightweight real-time analysis and processing performed in parallel on audio and video frame data acquired synchronously from the same source. They are used to extract key information required for translation tasks and region encoding before encoding, thereby shortening the overall processing latency. The lightweight face detection network is a face detection model built on a deep convolutional neural network. It is used to quickly locate the speaker's facial region in the original video frame data and output the bounding box coordinates. This model is pre-trained on a public face detection dataset and pruned and quantized. The number of parameters is controlled within the millions, and real-time detection can be achieved on a general-purpose CPU. The lip feature point regression model is a lightweight regression model built on a facial key point detection algorithm. The algorithm accurately extracts the coordinates of key points on the inner and outer contours of the lips within the located facial region. The input is a facial region image, and the output is the two-dimensional coordinates of the lip key points with a preset number of key points, typically 20. The lip grayscale image sequence is an image sequence composed of multiple consecutive frames of lip region grayscale images arranged in chronological order. A single frame image is a fixed-size image obtained by cropping and grayscale conversion based on the minimum bounding rectangle of the lip key points. The facial bounding box coordinates are rectangular coordinate parameters representing the position and size of the speaker's facial region in the original video frame, expressed in two-dimensional coordinates of the upper left and lower right corners. These coordinates are used for subsequent adjustment of the region of interest quantization parameters and lip region cropping and positioning in video encoding.

[0076] The speech activity detection algorithm is an endpoint detection algorithm based on the joint decision of short-time energy and spectral flatness. It is used to determine whether the original audio data contains valid speech frame by frame, and to distinguish between speech frames and silence frames. Its preset decision thresholds include a short-time energy threshold and a spectral flatness threshold. The short-time energy threshold is adaptively updated according to the background noise level and is usually set to -40dB. The spectral flatness threshold is usually set to 0.7. The speech frame label sequence containing speech frames and silence frames is a binary label sequence corresponding to the original audio data frame by frame. Speech frames are labeled as 1 and silence frames are labeled as 0. It is used to guide the dynamic allocation of the bitrate in subsequent audio encoding. Timestamp alignment and data encapsulation are processes that perform time-series matching and structured packaging of various heterogeneous data based on the unified network time protocol timestamp. This is used to align and integrate various types of data obtained from collection and analysis according to the time dimension to ensure the temporal consistency of subsequent layered coding. The preprocessed data packet is a standardized processing unit formed after time alignment and encapsulation. It contains the original audio data, original video frame data, lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence corresponding to the same time window, which serve as the unified input for subsequent independent and scalable layered coding.

[0077] Specifically, such as Figure 2 As shown, raw audio data and raw video frame data are first acquired using a microphone and a camera, respectively. Based on the Network Time Protocol (NTP) clock, a common timestamp is assigned to each frame of audio and video data. The sampling rate of the raw audio data is denoted as [missing information]. In this embodiment Take 16kHz, and record the duration of each frame as... In this embodiment Take 20ms, which corresponds to 50 audio processing frames per second; the frame rate of the original video frame data is denoted as... In this embodiment Take 25fps, and record the duration of each frame as . In this embodiment A time interval of 40ms is used, which corresponds to 25 video processing frames per second. The sampling rate of the raw audio data is determined based on speech intelligibility requirements, the acoustic feature extraction accuracy requirements of the automatic speech recognition task, and the audio encoding bitrate budget in low-bandwidth cross-border scenarios. The frame rate of the raw video frame data is determined based on the smoothness requirements of real-time audio and video communication, the accuracy requirements for capturing lip movement temporal changes, and the overall bitrate constraints of video encoding. After data acquisition and timestamp marking, video-side analysis and audio-side detection are performed in parallel: On the video side, the raw video frame data is input into a lightweight face detection network. This network adopts a single-stage detection architecture, containing a 3-layer depthwise separable convolutional backbone network and a 1-layer detection head. The input is a single frame of raw video image, and the output is the coordinates of the facial bounding box. ,in The coordinates of the top-left corner of the face bounding box are shown in pixels. The coordinates of the bottom right corner of the facial bounding box are given. The image region within the facial bounding box is input into the lip feature point regression model. This model employs a two-stage convolutional cascade structure and is pre-trained on a public facial landmark dataset. During training, the mean squared error loss function is used, and the initial learning rate is set to [value missing]. Batch size set to 32, iteration rounds set to 100, output The coordinates of key points on the lips, among which The number of key points is determined based on the required accuracy of the inner and outer contours of the lips, the computational latency requirements for model inference, and the computing power of the terminal device, with a range of values ​​ranging from [value missing]. In this embodiment, the number of key points is 20; the minimum bounding rectangle is calculated based on all lip key points, and the bounding rectangle is adjusted according to the expansion ratio. Expand the cropping area outwards, with the expansion ratio being... The range of values ​​is In this embodiment, the expansion ratio is set to 1.2, and the cropped image is converted into a single-channel grayscale image and scaled to a fixed resolution. In this embodiment, 64×64 pixels are selected, and the lip grayscale image sequence is obtained after frame-by-frame processing. The width in pixels of the grayscale image of the lips. This represents the height in pixels of the grayscale image of the lips. On the audio side, short-time energy and spectral flatness are calculated frame-by-frame from the original audio data. The calculation of short-time energy is based on the frame energy calculation formula in existing digital signal processing, and the formula is as follows: In the formula For the first The short-time energy of frame audio, For the first The first frame The amplitude of each sampling point In this embodiment, the number of sampling points per frame is... This corresponds to a 20ms frame length at a 16kHz sampling rate; the calculation of spectral flatness is based on the existing formula for the ratio of the geometric mean to the arithmetic mean of the power spectrum, and the calculation formula is as follows: In the formula For the first Spectral flatness of frame audio For the first Frame audio number Power spectrum values ​​at each frequency point The total number of frequency points, This is the chain multiplication operator, representing multiplication by index. From 0 to Perform a multiplication operation on all power spectral values ​​within the range; combine short-time energy values. Compared with the preset short-time energy threshold Spectral flatness Compared with the preset spectral flatness threshold Compare them separately, when and When a frame is identified as a speech frame, it is marked as 1; otherwise, it is identified as a silence frame and marked as 0. This process is repeated frame by frame to obtain a speech frame label sequence, where the short-time energy threshold is... The threshold values ​​are determined based on the background noise baseline energy of the acquisition environment, the average energy amplitude of the speech signal, and the false negative rate of speech activity detection. In this embodiment, -40dB is used as the spectral flatness threshold. Based on the differences in the spectral distribution characteristics of speech and background noise, the granularity of frequency point division, and the robustness requirements of the detection algorithm, this embodiment uses 0.7.

[0078] Then, based on the network time protocol timestamps carried by each data item, timestamp alignment and data encapsulation are performed. Using the video frame timestamp as a reference, the original audio data, lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence within the corresponding time window are matched and associated to ensure that all data within the same preprocessing data packet corresponds to the same time interval. During encapsulation, the data is organized according to a fixed structure of "timestamp, original audio data, original video frame data, lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence segment". Each data segment is labeled with its corresponding data length and time offset, forming a standardized preprocessing data packet, which serves as the input for subsequent independent and scalable layered coding. In this embodiment, the values ​​of sampling rate, frame rate, frame length, number of keypoints, expansion ratio, image resolution, and decision threshold are only examples. Those skilled in the art can set these values ​​according to actual scenarios and hardware conditions. This embodiment does not impose any limitations on these values.

[0079] This embodiment ensures the temporal consistency of audio and video data through synchronous acquisition using the same source clock, reduces preprocessing latency by performing video lip region extraction and audio speech activity detection in parallel, achieves accurate speech endpoint detection through joint decision based on short-time energy and spectral flatness, and forms standardized preprocessing data packets through unified timestamp alignment and structured encapsulation. This realizes a low-complexity, low-latency audio and video content preprocessing workflow, improves the accuracy of lip region localization and the noise resistance of speech activity detection, and provides reliable standardized input for subsequent layered coding and cross-modal fusion translation.

[0080] Furthermore, this embodiment provides a step of obtaining a layered bitstream set based on preprocessed data packets through independent scalable layered coding, including:

[0081] Based on the original audio data and original video frame data, an audio layered bitstream containing the basic audio layer bitstream and the audio enhancement layer bitstream is obtained by disabling the inter-frame dependency audio encoder for layered encoding.

[0082] Based on the original video frame data and facial bounding box coordinates, a scalable video coding method with reduced quantization parameters for the facial region is used to obtain a video layered bitstream containing a basic video layer bitstream and a video enhancement layer bitstream.

[0083] Based on the lip grayscale image sequence, a visual-speech layer data packet sequence containing the visual-speech base layer bitstream and the visual-speech enhancement layer bitstream is obtained through a visual-speech feature extraction network and two-level residual vector quantization hierarchical coding; specifically including:

[0084] Based on the lip grayscale image sequence, a visual-speech feature vector sequence is obtained through a visual-speech feature extraction network cascaded with a 3D spatiotemporal convolutional network and a bidirectional long short-term memory network. Based on the visual-speech feature vector sequence, coarse quantization is performed through first-level residual vector quantization to obtain the visual-speech base layer bitstream and the first-level quantization residual vector sequence. Based on the first-level quantization residual vector sequence, fine quantization is performed through second-level residual vector quantization to obtain the visual-speech enhancement layer bitstream. Based on the visual-speech base layer bitstream and the visual-speech enhancement layer bitstream, the visual-speech layer data packet sequence is obtained by aligning the timestamps to the covered audio frame range.

[0085] Based on the audio layered bitstream, video layered bitstream, and visual-speech layer data packet sequences, a set of layered bitstreams is obtained by organizing and encapsulating them according to type.

[0086] The audio encoder layered encoding with inter-frame dependency disabled is a layered encoding process based on the Opus open-source audio encoder. It achieves independent decoding of a single frame by disabling long-term inter-frame prediction dependencies. This process generates an audio layered bitstream that can be independently discarded at each level and decoded without dependency on a single frame, adapting to the high packet loss and bandwidth fluctuation characteristics of cross-border links. The audio layered bitstream is a two-level scalable bitstream set obtained after layered encoding of the original audio data, including the audio base layer bitstream and the audio enhancement layer bitstream. The audio base layer bitstream is a basic bitstream encoded at an extremely low bitrate, preserving speech intelligibility and the minimum acoustic features required for automatic speech recognition. It can be decoded completely independently without relying on previous and subsequent frames or enhancement layer information. The audio enhancement layer bitstream is an enhancement bitstream that supplements high-frequency components and sound quality details of the speech. It requires the corresponding frame's audio base layer bitstream to be decoded and is used to improve the naturalness and clarity of the speech.

[0087] Scalable video coding for facial regions, which reduces quantization parameters, is a video coding process based on the H.264 / SVC Scalable Video Coding Standard and combines macroblock quantization parameters adjusted according to facial regions of interest. This prioritizes visual clarity for facial and lip regions under overall low bitrate constraints. The layered video bitstream is a two-level scalable bitstream set obtained after scalable coding of the original video frame data, including a base layer bitstream and a enhancement layer bitstream. The base layer bitstream is a low-bitrate video bitstream encoded at a base resolution and frame rate, which can be decoded independently and provides basic visual information. The enhancement layer bitstream is a set of enhanced bitstreams used to restore the full frame rate, original resolution, and background texture quality. It requires the base layer bitstream for decoding and includes three sub-layers: temporal enhancement, spatial enhancement, and quality enhancement.

[0088] The visual-speech feature extraction network is a lightweight feature extraction model composed of a 3D spatiotemporal convolutional layer and a bidirectional long short-term memory network. It is used to extract high-dimensional discriminative features representing articulation actions from a sequence of grayscale lip images. The 3D spatiotemporal convolutional layer is a neural network layer that performs convolution operations simultaneously in the spatial and temporal dimensions, used to capture the spatial texture and short-term motion features of the lip region. The bidirectional long short-term memory network is a recurrent neural network layer with bidirectional temporal modeling capabilities, used to model the temporal contextual dependencies of long-term articulation actions. The visual-speech feature vector sequence is a high-dimensional feature vector generated frame-by-frame from the corresponding grayscale lip images. A sequence of dimensional feature vectors arranged in chronological order serves as the input for subsequent quantization encoding. Two-level residual vector quantization hierarchical encoding performs coarse and fine two-level residual quantization compression on the visual-speech feature vectors to generate a hierarchically decodeable, low-bitrate visual-speech hierarchical bitstream, including first-level and second-level residual vector quantization. The first-level residual vector quantization performs coarse-grained quantization on the original feature vectors to generate an independently decodeable basic feature bitstream, where the original feature vectors are the visual-speech feature vector sequence. The visual-speech basic layer bitstream is the input of the first-level residual vector quantization. The output codeword index sequence can be independently decoded to recover coarse articulation action features and capture major articulation action category information; the first-level quantization residual vector sequence is the difference sequence between the original feature vector and the feature vector reconstructed by the first-level quantization, containing residual information for fine lip shape differentiation; the second-level residual vector quantization is a fine-grained quantization process performed on the first-level quantization residual vector sequence to generate an enhanced codestream that supplements fine features; the visual-speech enhancement layer codestream is the codeword index sequence output by the second-level quantization, which needs to be superimposed and decoded based on the visual-speech base layer codestream to supplement fine lip shape differentiation information; the timestamp is aligned to the... The audio frame coverage is the process of mapping the timestamps of visual and speech feature frames to the corresponding audio frame time intervals within the time span, which is used to provide a timing alignment basis for cross-modal fusion at the receiving end; the visual and speech layer data packet sequence is a data packet sequence composed of visual and speech layered bitstreams carrying timestamp information in chronological order, including the visual and speech base layer bitstream and the visual and speech enhancement layer bitstream; the visual and speech base layer bitstream is coarse quantization index data encapsulated in the data packet and has independent decoding capability; the visual and speech enhancement layer bitstream is fine quantization residual index data encapsulated in the data packet and needs to rely on the base layer for joint decoding.

[0089] Organizing and encapsulating by type involves classifying, organizing, and standardizing the layered bitstreams of three different modalities according to their layer type, forming a set of bitstreams with a unified structure and clear hierarchy, which facilitates subsequent selection by the transmission scheduling module according to priority. The layered bitstream set is a complete layered bitstream whole formed after classification and encapsulation, containing bitstream data of six layers: audio base layer bitstream, audio enhancement layer bitstream, video base layer bitstream, video enhancement layer bitstream, visual and speech base layer bitstream, and visual and speech enhancement layer bitstream.

[0090] Specifically, such as Figure 3 As shown, the original audio data is first extracted from the preprocessed data packet, and then layered encoding with inter-frame dependencies disabled is performed using the layered encoding mode of the Opus encoder. During encoding, the long-term inter-frame prediction dependency option of the encoder is disabled, ensuring that the encoding result of each audio frame is completely independent, with no cross-frame prediction dependency, thus avoiding decoding failure of subsequent frames due to single-frame packet loss. The target bitrate of the audio base layer bitstream is denoted as... Based on the speech intelligibility threshold, the minimum accuracy requirement for automatic speech recognition, and the basic bandwidth budget for cross-border links, the target bitrate for the audio base layer bitstream in this embodiment is set to 8 kbps. This bitrate ensures that the speech content is intelligible and retains the core acoustic features required for automatic speech recognition. The target bitrate for the audio enhancement layer bitstream is denoted as... Based on the sound quality improvement requirements and the remaining bandwidth budget, the target bit rate of the audio enhancement layer bitstream in this embodiment is 16kbps, which is used to supplement the high-frequency components of speech and bandwidth expansion information to improve the naturalness of hearing; after encoding, an audio layered bitstream containing the audio basic layer bitstream and the audio enhancement layer bitstream is obtained.

[0091] Then, the original video frame data and facial bounding box coordinates are extracted from the preprocessed data packet, and scalable video coding to reduce the quantization parameters of the facial region is performed using the H.264 / SVC scalable video coding standard; the base resolution of the video base layer is configured as follows: The base resolution of the video base layer is determined based on basic visual experience requirements and bitrate constraints. In this embodiment, the base resolution of the video base layer is 320×180 pixels, and the base frame rate of the video base layer is denoted as [missing information]. The base frame rate of the video base layer is determined based on the minimum requirements for smoothness and bitrate constraints. In this embodiment, the base frame rate of the video base layer is 10fps, and the video base layer can be decoded completely independently. During the encoding process, a facial region of interest strategy is introduced in the macroblock-level quantization parameter decision stage: all encoded macroblocks in the current frame are traversed, and it is determined whether the macroblock falls within the facial region based on the facial bounding box coordinates. If the macroblock falls completely or partially within the facial region, the quantization parameter of the macroblock is reduced. The quantization parameter reduces the amplitude. Based on the priority of facial region sharpness and the overall bitrate control target, in this embodiment, the quantization parameter is reduced by an amplitude of 7 to force the preservation of image details in the facial and lip regions. The video enhancement layer is set with three sub-layers: temporal enhancement layer, spatial enhancement layer, and quality enhancement layer, which are used to restore the original 25fps frame rate, the original acquisition resolution, and the complete image quality of the background area, respectively. All enhancement layers rely on the decoding of the video base layer bitstream. After encoding, a layered video bitstream containing the video base layer bitstream and the video enhancement layer bitstream is obtained.

[0092] Furthermore, a grayscale image sequence of the lips is extracted from the preprocessed data packet, and visual-speech feature extraction, two-level residual vector quantization, and time-aligned encapsulation are performed sequentially to obtain the visual-speech layer data packet sequence. Specifically, visual-speech feature extraction is first performed, and the continuous... A grayscale image of the lips is used as input to a visual-speech feature extraction network, where the input frame length is... Based on the short-term correlation of the vocalization action and the model inference delay, this embodiment determines... Sixteen frames were captured, corresponding to a time span of approximately 640ms. The network's stride was 1 frame, consistent with the video frame rate. The visual-speech feature extraction network employed a cascaded structure of 3D spatiotemporal convolution, bidirectional long short-term memory (LSTM) network, and fully connected layers. The 3D spatiotemporal convolution consisted of three layers, each with a kernel size of 3×3×3, a stride of 1×1×1, and channel numbers of 32, 64, and 128 respectively, used to extract spatial details and motion features layer by layer. The bidirectional LSM network had a hidden layer dimension of 128, used for bidirectional modeling of temporal context. Finally, the output was projected through the fully connected layer. dimensional feature vectors, where the feature dimension is 1. Based on the feature discrimination requirements and quantization bitrate constraints, the feature dimension is set to 128 in this embodiment. The visual-speech feature extraction network is pre-trained on a cross-language lip-reading public dataset. During training, a joint loss function combining mean squared error loss and contrastive loss is used, and the initial learning rate is set to... The batch size is set to 16, and the iteration rounds are set to 50, enabling cross-language generalization. The output of the visual-speech feature extraction network is a 128-dimensional feature vector that is time-aligned with the center frame of the input sequence. After frame-by-frame sliding processing, a visual-speech feature vector sequence is obtained. Then, the first-level residual vector quantization is performed on each frame of the visual-speech feature vector. Through the first-level codebook Perform nearest neighbor search quantization, where the first-level codebook The number of codewords is recorded as Based on the coarse-grained classification accuracy and bitrate budget, this embodiment determines... The index length is 64, corresponding to 6 bits. The codebook is trained using the existing LBG vector quantization algorithm based on a cross-language lip-reading dataset. After quantization, the corresponding codeword index sequence is output, which is the basic layer codestream of the visual speech, with a bitrate of approximately 6 bits × 25 fps = 150 bps. At the same time, the difference between the original feature vector and the quantized reconstructed vector is calculated to obtain the first-level quantization residual vector sequence. Furthermore, a second-level residual vector quantization is performed, converting the first-level quantized residual vector sequence... Input the second-level codebook Perform fine-grained metric quantization, where the second-level codebook The number of codewords is recorded as In this embodiment A value of 64 is chosen, corresponding to a 6-bit index length, which is also obtained through training on the residual dataset using the LBG algorithm. After quantization, the corresponding residual codeword index sequence is output, which is the visual-speech enhancement layer bitstream, with a bitrate of approximately 150bps. During decoding, it needs to be superimposed with the first-level quantization result to recover the complete features. Finally, timestamp-aligned encapsulation is performed. Since the visual-speech features of a single frame correspond to a 40ms time span, which exactly covers two consecutive 20ms audio frames, the start and end audio frame numbers corresponding to the basic visual-speech layer bitstream and the visual-speech enhancement layer bitstream of each frame are written into the data packet header to explicitly indicate the audio frame time range covered by the visual-speech frame. After encapsulation frame by frame, a time-aligned visual-speech layer data packet sequence is obtained.

[0093] Finally, the audio layered bitstream, video layered bitstream, and visual / speech layer data packet sequences are organized and encapsulated by type. During encapsulation, data is categorized and grouped according to modality and layer type, and a unified layer identifier field, timestamp field, and data length field are added to the data packets of each layer. All data packets follow the same header format. After being organized in hierarchical order, a set of layered bitstreams with standardized structure and independently schedulable layers is obtained, which serves as the input data source for subsequent priority-based layered transmission scheduling. In this embodiment, the values ​​of audio bitrate, video resolution and frame rate, quantization parameter reduction magnitude, number of feature input frames, feature dimension, and codebook size are only examples. Those skilled in the art can set these values ​​according to actual scenarios and performance requirements, and this embodiment does not impose any limitations on them.

[0094] This embodiment improves the robustness of audio decoding in high packet loss scenarios by disabling long-term prediction in audio layered coding. Video scalable coding, adjusted by facial region of interest quantization parameters, prioritizes the clarity of key visual regions at low bitrates. It constructs an ultra-low bitrate, independently decodeable layered visual-speech bitstream through visual-speech feature extraction independent of video coding and two-level residual vector quantization. By standardizing and encapsulating by type to form a clearly defined set of layered bitstreams, this embodiment achieves a completely decoupled, independent, and scalable coding architecture for audio, video, and visual-speech bitstreams. This enhances the flexibility of bitstream scheduling in cross-border low-bandwidth fluctuation scenarios and the cross-modal redundancy guarantee capability of translation tasks, while controlling overall coding complexity and ensuring both subjective audio-visual quality and the reliability of translation input.

[0095] Furthermore, this embodiment provides a step for obtaining a set of data packets to be sent based on a hierarchical bitstream set and network feedback information through priority-based hierarchical transmission scheduling, including:

[0096] Based on network feedback information, a congestion control algorithm that jointly estimates the delay gradient and packet loss rate is used to obtain an estimate of the available bandwidth.

[0097] Based on the hierarchical bitstream set, priority sending tags are obtained through translation task priority classification rules;

[0098] Based on priority sending tags and available bandwidth estimates, a set of data packets is obtained through priority-based hierarchical transmission scheduling.

[0099] Among them, the network feedback information is the cross-border link status data periodically transmitted back by the receiving end through the real-time transmission control protocol, including the arrival timestamp and packet loss rate periodically fed back by the receiving end, which is used to provide the link status basis for the bandwidth estimation of the sending end; the congestion control algorithm for joint estimation of delay gradient and packet loss rate is an algorithm based on the GCC (GNU Compiler Collection) congestion control framework, which combines the one-way delay change trend and packet loss level to jointly determine the link congestion status and estimate the available bandwidth, which is used to sense the bandwidth fluctuation of the cross-border link in real time and output the maximum transmission rate that can be carried at present; the available bandwidth estimate is the maximum data transmission rate that the current cross-border link can stably carry under the premise of ensuring low latency.

[0100] The translation task priority classification rule is based on the minimum availability of the translation task and prioritizes each layer of bitstream according to its importance in supporting the translation function. This rule is used to assign differentiated transmission priorities to each layer of bitstream to ensure that the core translation function is not interrupted under extreme bandwidth conditions. The priority transmission mark is a priority identifier field attached to the header of each bitstream data packet to identify the transmission priority of the layer to which the data packet belongs, so as to quickly identify and filter.

[0101] Priority-based hierarchical transmission scheduling is an adaptive scheduling process that discards data packets layer by layer from low to high priority based on available bandwidth constraints and priority marking. It is used to dynamically adjust the combination of transmitted code streams when bandwidth fluctuates, so as to maximize the transmission of translation-related core data. The set of transmitted data packets is the whole set of data packets to be transmitted after priority scheduling and the total transmission rate does not exceed the estimated value of available bandwidth. Under any bandwidth conditions, high-priority translation core layer data is reserved first.

[0102] Specifically, such as Figure 4 As shown, the receiving end first periodically transmits network feedback information back to the sending end via the real-time transmission control protocol. The feedback period is denoted as . The value is determined based on the link delay fluctuation characteristics and bandwidth estimation sensitivity requirements, and is typically set to 200ms. The feedback content includes the arrival timestamp of each received data packet and packet loss statistics within the statistical period. Based on the received network feedback information, the sending end calculates the available bandwidth estimate using a congestion control algorithm that jointly estimates the delay gradient and packet loss rate. On the one hand, the one-way delay of each data packet is calculated based on the sending timestamp and the arrival timestamp. The delay gradient is calculated by measuring the delay changes between adjacent statistical periods. When the delay gradient continues to increase, it is determined that the link is in an upward trend of congestion, and the bandwidth estimate is adjusted downward accordingly. On the other hand, the packet loss rate within the statistical period is calculated based on the feedback packet loss statistics. Its calculation formula is In the formula To count the number of lost data packets within the statistical period. The system calculates the total number of data packets sent within a statistical period. When the packet loss rate exceeds a preset congestion threshold, the link is considered congested, and the bandwidth estimate is adjusted downwards accordingly. When the packet loss rate is less than a preset idle packet loss threshold and the latency gradient is stable, the link bandwidth is considered to be abundant, and the bandwidth estimate is adjusted upwards accordingly. Finally, the system outputs a real-time updated estimated available bandwidth. As the basis for bandwidth constraints in transmission scheduling, the preset congestion packet loss threshold is determined based on the acceptable packet loss level of the link and the robustness of translation recognition, and is usually set to 5%. The preset idle packet loss threshold is determined based on the statistical characteristics of the link's idle state, and is usually set to 0.5%.

[0103] Then, based on all data packets in the hierarchical bitstream set, a priority sending tag is added to each data packet according to the translation task priority classification rule. The translation task priority classification rule divides the six-level bitstream into six priority levels according to their importance to the translation function, from high to low. The priority level is denoted as... The value is an integer from 1 to 6, with smaller values ​​indicating higher priority: the priority corresponding to the audio base layer bitstream. It provides the necessary acoustic foundation information for the translation task and enjoys the highest priority; the visual-speech basic layer code streams have corresponding priorities. It provides cross-modal redundancy information for translation tasks and has the second highest priority; the video basic layer bitstream has the corresponding priority. Used to maintain the continuity of the basic visual image; the audio enhancement layer bitstream corresponds to a priority level. Used to enhance the auditory sound quality experience; the visual and speech enhancement layer bitstream has a corresponding priority. Used to supplement fine lip-sync features; the video enhancement layer bitstream corresponds to a higher priority. Used to restore complete image quality and frame rate, with the lowest priority; iterates through all data packets in the layered bitstream set, writes the corresponding priority level value in the priority field of the data packet header according to its layer, and obtains all data packets to be sent carrying priority sending tags.

[0104] Finally, priority-based hierarchical transmission scheduling is executed with a fixed scheduling period, where the scheduling period is denoted as . The value is determined based on the real-time requirements of system scheduling and the granularity of audio frame processing, and is usually set to 20ms, consistent with the audio processing frame length. Within each scheduling cycle, the total transmission rate of all data packets to be sent is first calculated. The total transmission rate Compared with available bandwidth estimates Comparison: If Then all data packets to be sent are included in the set of data packets to be sent; if Then from the lowest priority level The corresponding data packets are started to be dropped. After all data packets at each level are dropped, the remaining total transmission rate is recalculated until the remaining total transmission rate is less than or equal to the estimated available bandwidth. Up to this point, the remaining data packets together constitute the set of data packets to be sent; the priority-based hierarchical transmission scheduling strategy ensures that, as bandwidth continues to decrease, high-priority translation core layer data is always retained first, and at least priority data is retained even in extremely low-bandwidth scenarios. and The corresponding audio base layer bitstream and visual-speech base layer bitstream ensure that the translation engine has the minimum usable input for both audio and visual modalities. In this embodiment, the values ​​for feedback period, packet loss threshold, priority level, and scheduling period are merely examples; those skilled in the art can set them according to actual scenarios and performance requirements, and this embodiment does not impose any limitations on them.

[0105] This embodiment achieves real-time and accurate perception of available bandwidth for cross-border links through the GCC congestion control algorithm based on joint estimation of delay gradient and packet loss rate. It establishes a hierarchical ranking system oriented towards translation experience by prioritizing the availability of translation tasks. It adaptively matches bandwidth fluctuations through a priority-based layer-by-layer discarding scheduling mechanism with fixed periods. It strengthens the availability baseline of translation functions through a dual-modal core layer guarantee strategy under extreme bandwidth conditions. This realizes refined adaptive transmission scheduling of bitstreams in low-bandwidth cross-border scenarios, improves the stability and packet loss resistance of translation services when network bandwidth fluctuates drastically, and prioritizes core translation functions under limited bandwidth resources.

[0106] Furthermore, this embodiment provides a step for obtaining a video frame sequence based on a set of transmitted data packets through layered decoding, time alignment, and cross-modal fusion translation, including:

[0107] Based on the transmitted data packet set, decoded audio PCM signals, decoded video frame sequences, and visual-speech feature sequences are obtained through layered decoding and time alignment; specifically including:

[0108] Based on the set of transmitted data packets, the decoded audio PCM signal is obtained through the Opus standard decoder; based on the set of transmitted data packets, the decoded video frame sequence is obtained through the H.264 scalable video decoder; based on the set of transmitted data packets, the decoded visual-speech feature vector sequence is obtained through the residual vector quantization decoder; based on the decoded visual-speech feature vector sequence and its audio frame time range information, the feature frame rate is extended to be consistent with the audio frame rate through linear interpolation to obtain the visual-speech feature sequence.

[0109] Based on decoded audio PCM signals and visual-speech feature sequences, target language translated text is obtained through acoustic feature extraction, robust audiovisual fusion recognition, and neural machine translation; specifically including:

[0110] Based on the decoded audio PCM signal, an acoustic feature sequence is obtained through Mel filter bank analysis; based on the acoustic feature sequence and the visual-speech feature sequence, an audiovisual fusion recognition model with a dual encoder plus fusion decoder architecture and a modality dropout strategy introduced during training is used to obtain the source language recognition text sequence; based on the source language recognition text sequence, the target language translation text is obtained through neural machine translation.

[0111] Based on the decoded video frame sequence, visual and speech feature sequence, and target language translated text, a lip-smoothing interpolation is performed on the missing frames using a lip-shape generation model to obtain the video frame sequence.

[0112] The layered decoding and time alignment process involves the receiver independently decoding each layer of the bitstream arriving via cross-border network transmission, and performing time-series calibration on the decoding results for different frame rates and modalities. This integrated process is used to recover the original multimodal data and establish a unified timing reference, ensuring the timing consistency of subsequent cross-modal fusion translation. The Opus standard decoder is a standard audio decoding module corresponding to the sender's Opus encoder, used to decode and reconstruct the layered audio bitstream to recover the time-domain audio signal. The decoded audio PCM signal is a pulse code modulation format digital audio signal obtained from the decoding and reconstruction, consistent with the original audio data format. The H.264 scalable video decoder is a standard scalable video decoding module corresponding to the sender's H.264 / SVC encoding, used to decode and reconstruct the layered video bitstream to recover the video frame. The decoded video frame sequence is a digital image frame sequence obtained from the decoding and reconstruction, the resolution and frame rate of which depend on the number of video layers arriving. The residual vector quantization decoder is a module corresponding to the sender's... The decoding module corresponding to the two-level residual vector quantization layered encoding is used to recover visual-speech feature vectors based on codeword indexes and source codebooks. The decoded visual-speech feature vector sequence is a time-series composition of the decoded high-dimensional visual-speech feature vectors, with a frame rate consistent with the original video frame rate. The audio frame time range information is the temporal annotation information encapsulated in the header of the visual-speech layer data packet, written according to the time alignment relationship during encoding at the sending end, used to indicate the audio frame time interval covered by a single frame of visual-speech features. Linear interpolation is a common temporal interpolation method in the field of digital signal processing, used to improve the frame rate of visual-speech features and achieve frame-by-frame alignment with audio frames. The feature frame rate is the number of frames per second of the decoded visual-speech feature vector sequence, determined based on the original video acquisition frame rate. The audio frame rate is the number of frames per second of audio processing, determined based on the audio sampling rate and single frame duration. The visual-speech feature sequence is a sequence of visual-speech feature vectors corresponding to audio frames after frame rate expansion and temporal alignment, serving as the input for subsequent cross-modal fusion recognition.

[0113] Acoustic feature extraction is the process of extracting frequency domain discriminative features suitable for speech recognition tasks from time-domain audio signals, used to convert the original time-domain audio into a feature representation that better conforms to the characteristics of human hearing; Mel filter bank analysis is a standard acoustic feature extraction method based on the Mel frequency scale, used to map the audio power spectrum into Mel domain filtering features; the acoustic feature sequence is a sequence of Mel frequency domain feature vectors corresponding to each frame, serving as the audio input for the speech recognition model; robust audiovisual fusion recognition is the process of simultaneously utilizing acoustic modal and visual-speech modal information to perform speech recognition, used to maintain stable recognition accuracy even when single-modal quality degrades due to bandwidth limitations; the dual encoder plus fusion decoder architecture separately processes acoustic modal and visual-speech modal information... A neural network structure that performs independent encoding and then performs fusion decoding through a cross-attention mechanism is used to achieve dynamic weighted fusion of bimodal information. The audiovisual fusion recognition model that introduces a modality dropout strategy during training is an audiovisual fusion speech recognition model that randomly masks single-modal inputs during the training phase to improve the model's adaptability to single-modal missing or degraded scenarios. The source language recognition text sequence is the segmented text sequence of the recognized source language, carrying corresponding timestamps. The neural machine translation is an end-to-end machine translation method based on deep neural networks, used to convert source language text into target language text. The target language translated text is the translated target language text content, carrying corresponding speech timestamps, used for subtitle presentation and subsequent lip-syncing reference.

[0114] The lip-sync generation model is a lightweight generative network driven by visual and speech features, used to generate lip region images at corresponding times based on the features; missing frames are time interval frames formed by missing video frames or insufficient frame rate due to network packet loss, failure of enhancement layers to arrive, etc.; lip smoothing interpolation is a process that uses the visual and speech features of adjacent frames and the lip-sync generation model to smoothly fill in the lip region of missing frames, which is used to improve the continuity of video frame rate and lip-sync effect; the video frame sequence is a complete and continuous video frame sequence obtained after decoding and reconstruction, missing frame filling and lip-sync processing, with lip movement effects aligned with audio and translated text.

[0115] Specifically, such as Figure 5As shown, the receiving end first receives the set of transmitted data packets arriving via the cross-border network, classifies and distributes them according to their hierarchical identifiers, performs parallel decoding, and then completes time alignment. The decoding process includes audio decoding, video decoding, visual-speech decoding, and time alignment. Audio decoding includes extracting the data packets corresponding to the basic audio layer bitstream from the transmitted data packet set. If audio enhancement layer bitstream data packets arrive, they are extracted synchronously and input into the Opus standard decoder. Since long-term prediction dependencies between frames are disabled during the encoding stage, each audio frame can be decoded independently, and single-frame packet loss will not cause subsequent frame decoding failures. The decoded output is a decoded audio PCM signal with the same sampling rate as the original acquisition. Video decoding includes extracting the data packets corresponding to the basic video layer bitstream from the transmitted data packet set, and inputting the arrived video enhancement layer bitstream data packets into the H.264 scalable video decoder. This decoder reconstructs the image level by level according to the arriving enhancement layers: when only the basic video layer arrives, it outputs a decoded video frame sequence with the basic resolution and basic frame rate; when the temporal enhancement layer, spatial enhancement layer, and quality enhancement layer arrive sequentially, the corresponding complete frame rate, original resolution, and complete image quality are restored, ultimately outputting a decoded video frame sequence. Visual-speech decoding includes: extracting data packets corresponding to the basic visual-speech layer codestream from the transmitted data packet set; if the visual-speech enhancement layer codestream data packet arrives, it is extracted synchronously and input into the residual vector quantization decoder. This decoder uses a first-level codebook and a second-level codebook completely identical to the one used by the transmitter. First, the first-level codebook is retrieved based on the basic layer codeword index to obtain a coarse-grained reconstructed feature vector. If the visual-speech enhancement layer arrives, the second-level codebook is retrieved based on the visual-speech enhancement layer codeword index to obtain the residual vector. The coarse-grained feature vector and the residual vector are superimposed to obtain a complete reconstructed feature vector. After frame-by-frame processing, a decoded visual-speech feature vector sequence is obtained, with a frame rate consistent with the original video frame rate of 25fps. Time alignment processing includes: reading the audio frame time range information from the header of each visual-speech data packet; using linear interpolation to extend the frame rate of the decoded visual-speech features in the decoded visual-speech feature vector sequence from 25fps to 50fps, consistent with the audio processing frame rate. The linear interpolation calculation is based on the first-order linear interpolation formula in existing digital signal processing, and its calculation formula is as follows: In the formula For the interpolated first Visual and speech feature vectors of a frame. and These are the original decoded visual and speech feature vectors of two adjacent frames. These are the interpolation weighting coefficients, with a value range of [value range missing]. The time position of the interpolated frame between two original feature frames is determined; after frame-by-frame interpolation is completed, a visual-speech feature sequence corresponding to each audio frame is obtained.

[0116] Then, cross-modal audiovisual fusion translation processing is performed, sequentially completing acoustic feature extraction, audiovisual fusion recognition, and neural machine translation. Acoustic feature extraction includes: performing Mel filter bank analysis on the decoded audio PCM signal, with a frame length consistent with the audio processing frame length of 20ms, extracting Mel filter bank features frame by frame; the number of Mel filter banks is denoted as... Based on the accuracy requirements and computational complexity constraints of speech recognition, the number of Mel filter banks in this embodiment is set to 40, meaning that each frame outputs a 40-dimensional acoustic feature vector. After processing frame by frame, an acoustic feature sequence is obtained, with a frame rate of 50fps, perfectly aligned with the visual-speech feature sequence. Robust audiovisual fusion recognition includes: an audiovisual fusion recognition model that synchronously inputs the acoustic feature sequence and the visual-speech feature sequence into training, incorporating a modality discarding strategy; this model adopts a Transformer architecture with dual encoders and a fusion decoder, wherein the audio encoder contains... The Transformer encoder layer is determined based on the acoustic modeling complexity and accuracy requirements; in this embodiment... The value is 6, and the input is a 40-dimensional acoustic feature sequence; the visual-speech encoder contains The Transformer encoder layer has 4 layers, determined by the complexity of visual feature modeling. The input is a 128-dimensional visual-speech feature sequence. The fusion decoder includes... The Transformer decoder layer, in this embodiment, has 8 layers. It uses a two-way cross-attention mechanism to focus on the outputs of the audio encoder and the visual-speech encoder respectively, dynamically fusing information from the two modalities at each decoding step. The audiovisual fusion recognition model is pre-trained on a multilingual audiovisual speech dataset. During the training phase, a random modality dropout strategy is adopted, with each training round using the modality dropout probability. Randomly set all audio or visual features to zero, where the modality dropout probability is... Based on robustness requirements and dual-modal accuracy, this embodiment... A value of 0.2 is used to enable the audiovisual fusion recognition model to simultaneously master three working modes: dual-modal fusion recognition, pure audio recognition, and pure visual lip reading. During the deployment phase, the audiovisual fusion recognition model automatically adapts to the actual arriving modal input: it outputs the highest recognition accuracy when both modalities arrive, and outputs the optimal recognition result under the current conditions when only a single modality arrives, ultimately outputting a source language recognition text sequence with timestamps. Neural machine translation includes: inputting the source language recognition text sequence into a standard neural machine translation engine, performing end-to-end text translation, and obtaining target language translated text with corresponding timestamps, which can be used for subtitle overlay and speech synthesis output.

[0117] Finally, video reconstruction and lip-sync processing are performed to obtain a complete video frame sequence: First, based on the timestamp information of the decoded video frame sequence, the location and number of missing video frames due to network packet loss or layer loss are counted; then, a pre-trained lightweight lip-sync generation model is used, driven by the visual-speech feature vectors at the corresponding moments of the missing frames, to generate lip region images at the corresponding moments; this lip-sync generation model adopts a lightweight convolutional generative network structure, with a 128-dimensional visual-speech feature vector as input and a lip image with the same size as the lip cropping region as output. It is pre-trained on a public lip-reading video dataset, using a loss function combining mean squared error loss and perceptual loss during training, with an initial learning rate of [value missing]. The batch size is set to 8, and the iteration rounds are set to 30, enabling real-time generation on a general-purpose CPU. For multiple consecutive missing frames, lip smoothing interpolation is performed based on the lip regions and corresponding visual-speech features of adjacent decoded frames. Weighted fusion ensures a smooth transition of lip movements in time, avoiding abrupt changes in the image. Finally, the generated lip region images are seamlessly fused to the corresponding facial positions of the decoded video frames using a Poisson fusion algorithm. After completing all missing frames, a video frame sequence with complete frame rate and synchronized lip movements with audio and translated text is obtained. In this embodiment, the values ​​of Mel filter bank dimension, Transformer model layer number, modality dropout probability, and lip generation model parameters are only examples. Those skilled in the art can set these values ​​according to actual scenarios and performance requirements. This embodiment does not impose any limitations on these values.

[0118] This embodiment avoids the transmission impact of single-level packet loss on other modalities by using an architecture that independently decodes three types of bitstreams. It achieves precise temporal alignment of visual speech features and audio features frame by frame through frame rate expansion using linear interpolation. It improves the robustness of recognition in single-modal degradation scenarios by using a dual-encoder fusion recognition model with a random modality drop strategy. It completes missing video frames and ensures lip-sync effect by using lip-sync generation driven by visual speech features and smooth interpolation. This achieves stable cross-modal translation output and image reconstruction in low-bandwidth, high-packet-loss cross-border scenarios, improves the accuracy of speech recognition translation and the visual coherence of video images under cross-border network conditions, and achieves the effect of smooth service quality degradation with bandwidth while ensuring that the core translation function is always available.

[0119] Example 2:

[0120] like Figure 6 As shown, this embodiment provides an audio and video layered coding translation system for low-bandwidth cross-border scenarios, including:

[0121] Data preprocessing module: used to acquire raw audio data and raw video frame data, and obtain preprocessed data packets through content analysis and preprocessing;

[0122] Layered coding module: Based on preprocessed data packets, it obtains a set of layered bitstreams through independent and scalable layered coding;

[0123] Transmission scheduling module: Based on the hierarchical bitstream set and network feedback information, it obtains the set of data packets to be sent through priority-based hierarchical transmission scheduling;

[0124] Fusion Translation Module: Based on the set of sent data packets, video frame sequences are obtained through layered decoding, time alignment, and cross-modal fusion translation.

[0125] The hierarchical coding module, transmission scheduling module, and fusion translation module are all located on the server. The server receives data transmitted from the acquisition devices and performs further analysis and processing. The data preprocessing module can be deployed on the acquisition terminal or integrated into the server side. The acquisition devices include a microphone for acquiring raw audio data and a camera for acquiring raw video frame data. The acquisition devices establish a data interaction channel with the server through a network transmission link, uploading the synchronously acquired raw audio and video data to the server for subsequent processing. The data preprocessing module performs parallel lip region extraction and speech activity detection on the synchronously acquired raw audio and video frame data, and outputs standardized preprocessed data packets after completing unified timestamp alignment and structured encapsulation. After receiving the preprocessed data packets, the hierarchical coding module performs parallel audio hierarchical coding with frame dependencies disabled, scalable video coding with facial region weighting, and independent visual-speech two-level residual vector quantization coding. This generates a hierarchical bitstream set containing an audio base layer, an audio enhancement layer, a video base layer, a video enhancement layer, a visual-speech base layer, and a visual-speech enhancement layer, constructing a scalable bitstream architecture where each layer can be decoded independently and can be discarded according to priority. After receiving the hierarchical bitstream set, the transmission scheduling module performs parallel processing based on... The receiving end periodically transmits network feedback information to estimate the available bandwidth of the cross-border link in real time. Priority sending tags are added to each layer of the bitstream according to a priority classification rule based on the minimum availability of the translation task. Data packets are then filtered and discarded layer by layer from low to high based on available bandwidth constraints, outputting a set of data packets adapted to the current bandwidth to ensure the transmission priority of the core bitstreams related to translation. The fusion translation module receives the set of data packets arriving via the cross-border network and sequentially performs independent decoding of the three types of bitstreams, cross-modal temporal alignment, audiovisual fusion speech recognition, and neural machine translation. It also performs lip-smoothing interpolation and lip-syncing processing on missing video frames caused by network packet loss, ultimately outputting a video frame sequence with synchronized audio, video, and translated subtitles. Each module is sequentially connected according to the data flow direction, working together to complete the full-link audio and video layered encoding translation processing in low-bandwidth cross-border scenarios.

[0126] In summary, this invention addresses typical scenarios of low bandwidth, high fluctuations, and high packet loss rates in cross-border links. It constructs three scalable encoding architectures using a layered visual-speech bitstream independent of traditional audio and video bitstreams. A priority transmission scheduling mechanism centered on translation task availability achieves differentiated bandwidth resource allocation. A cross-modal audiovisual fusion translation model with random modality dropout training enhances recognition stability in single-modal degradation scenarios. Lip-sync interpolation driven by visual-speech features ensures image continuity and lip-sync synchronization. Targeted adaptation and optimization across the entire translation chain—encoding, transmission, decoding, and translation—ensures the availability of core translation functions in extremely bandwidth-constrained scenarios. This improves the anti-interference capability and translation accuracy of cross-border audio and video translation services under severe network fluctuations, while also considering the subjective experience of audio and video images and the overall low complexity and feasibility of the system. It can be applied to various low-bandwidth real-time audio and video translation scenarios such as cross-border remote conferencing, cross-border online education, and cross-border emergency communication.

[0127] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A layered audio-video coding translation method for low-bandwidth cross-border scenarios, characterized in that, include: The raw audio data and raw video frame data are acquired, and preprocessed data packets are obtained through content analysis and preprocessing. Based on the preprocessed data packets, a set of layered bitstreams is obtained through independent scalable layered coding. Based on the hierarchical bitstream set and network feedback information, the set of data packets to be sent is obtained through priority-based hierarchical transmission scheduling. Based on the set of transmitted data packets, a video frame sequence is obtained through layered decoding, time alignment, and cross-modal fusion translation.

2. The audio and video layered coding translation method for low-bandwidth cross-border scenarios according to claim 1, characterized in that, The raw audio data and raw video frame data are acquired, and preprocessed data packets are obtained through content analysis and preprocessing, including: Acquire raw audio data and raw video frame data; Based on the original audio data and original video frame data, parallel video analysis and audio detection are used to obtain the lip grayscale image sequence, facial bounding box coordinates and speech frame marker sequence. Based on the original audio data, original video frame data, lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence, a preprocessed data packet is obtained through timestamp alignment and data encapsulation.

3. The audio and video layered coding translation method for low-bandwidth cross-border scenarios according to claim 2, characterized in that, Based on the original audio data and original video frame data, parallel video analysis and audio detection were used to obtain the lip grayscale image sequence, facial bounding box coordinates, and speech frame marker sequence, including: Based on the original video frame data, a lightweight face detection network and a lip feature point regression model are used to obtain the lip grayscale image sequence and facial bounding box coordinates. Based on the original audio data, a speech frame marker sequence containing speech frames and silence frames is obtained through a speech activity detection algorithm.

4. The audio and video layered coding translation method for low-bandwidth cross-border scenarios according to claim 1, characterized in that, Based on the preprocessed data packets, a set of layered bitstreams is obtained through independent scalable layered coding, including: Based on the original audio data and original video frame data, an audio layered bitstream containing the basic audio layer bitstream and the audio enhancement layer bitstream is obtained by disabling the inter-frame dependency audio encoder for layered encoding. Based on the original video frame data and facial bounding box coordinates, a scalable video coding method with reduced quantization parameters for the facial region is used to obtain a video layered bitstream containing a basic video layer bitstream and a video enhancement layer bitstream. Based on the grayscale image sequence of the lips, a visual speech layer data packet sequence containing the basic visual speech layer code stream and the visual speech enhancement layer code stream is obtained by using a visual speech feature extraction network and a two-level residual vector quantization hierarchical coding. Based on the audio layered bitstream, video layered bitstream, and visual-speech layer data packet sequences, a set of layered bitstreams is obtained by organizing and encapsulating them according to type.

5. The audio and video layered coding translation method for low-bandwidth cross-border scenarios according to claim 4, characterized in that, Based on the lip grayscale image sequence, a visual-speech layer data packet sequence containing the visual-speech base layer bitstream and the visual-speech enhancement layer bitstream is obtained through a visual-speech feature extraction network and two-level residual vector quantization hierarchical coding, including: Based on the grayscale image sequence of the lips, a visual speech feature vector sequence is obtained through a visual speech feature extraction network cascaded with a three-dimensional spatiotemporal convolutional network and a bidirectional long short-term memory network. Based on the visual speech feature vector sequence, coarse quantization is performed through first-level residual vector quantization to obtain the visual speech basic layer code stream and the first-level quantization residual vector sequence; Based on the first-level quantized residual vector sequence, the visual-speech enhancement layer bitstream is obtained by fine quantization through the second-level residual vector quantization. Based on the basic visual-speech layer bitstream and the visual-speech enhancement layer bitstream, the visual-speech layer data packet sequence is obtained by aligning the timestamps to the range of covered audio frames.

6. The audio and video layered coding translation method for low-bandwidth cross-border scenarios according to claim 1, characterized in that, Based on the hierarchical bitstream set and network feedback information, a set of data packets to be sent is obtained through priority-based hierarchical transmission scheduling, including: Based on network feedback information, a congestion control algorithm that jointly estimates the delay gradient and packet loss rate is used to obtain an estimate of the available bandwidth. Based on the hierarchical bitstream set, priority sending tags are obtained through translation task priority classification rules; Based on priority sending tags and available bandwidth estimates, a set of data packets is obtained through priority-based hierarchical transmission scheduling.

7. The audio and video layered coding translation method for low-bandwidth cross-border scenarios according to claim 1, characterized in that, Based on the set of transmitted data packets, a video frame sequence is obtained through layered decoding, time alignment, and cross-modal fusion translation, including: Based on the set of transmitted data packets, the decoded audio PCM signal, decoded video frame sequence, and visual-speech feature sequence are obtained through layered decoding and time alignment. Based on decoded audio PCM signals and visual-speech feature sequences, target language translated text is obtained through acoustic feature extraction, robust audio-visual fusion recognition, and neural machine translation. Based on the decoded video frame sequence, visual and speech feature sequence, and target language translated text, a lip-smoothing interpolation is performed on the missing frames using a lip-shape generation model to obtain the video frame sequence.

8. The audio and video layered coding translation method for low-bandwidth cross-border scenarios according to claim 7, characterized in that, Based on the set of transmitted data packets, the decoded audio PCM signal, decoded video frame sequence, and visual-speech feature sequence are obtained through layered decoding and time alignment, including: Based on the set of transmitted data packets, the decoded audio PCM signal is obtained through the Opus standard decoder; Based on the set of transmitted data packets, the sequence of decoded video frames is obtained through the H.264 scalable video decoder. Based on the set of transmitted data packets, the decoded visual-speech feature vector sequence is obtained through a residual vector quantization decoder. Based on the decoded visual-speech feature vector sequence and its audio frame time range information, the feature frame rate is extended to match the audio frame rate through linear interpolation to obtain the visual-speech feature sequence.

9. The audio and video layered coding translation method for low-bandwidth cross-border scenarios according to claim 7, characterized in that, Based on decoded audio PCM signals and visual-speech feature sequences, target language translated text is obtained through acoustic feature extraction, robust audiovisual fusion recognition, and neural machine translation, including: Based on the decoded audio PCM signal, acoustic feature sequences are obtained through Mel filter bank analysis. Based on acoustic feature sequences and visual-speech feature sequences, an audio-visual fusion recognition model with a dual encoder plus fusion decoder architecture and a modality dropout strategy introduced during training is used to obtain the source language recognition text sequence. Based on the source language, the text sequence is identified, and the target language translation text is obtained through neural machine translation.

10. A layered audio-visual coding translation system for low-bandwidth cross-border scenarios, used to implement the layered audio-visual coding translation method for low-bandwidth cross-border scenarios as described in any one of claims 1-9, characterized in that, The system includes: The data preprocessing module is used to acquire raw audio data and raw video frame data, and obtain preprocessed data packets through content analysis and preprocessing. The layered encoding module obtains a set of layered bitstreams based on the preprocessed data packets through independent and scalable layered encoding; The transmission scheduling module obtains the set of data packets to be sent through priority-based hierarchical transmission scheduling based on the hierarchical code stream set and network feedback information. The fusion translation module obtains a video frame sequence based on the set of sent data packets through layered decoding, time alignment, and cross-modal fusion translation.

Images