Video conference audio and picture synchronization method and system

Abstract

The invention relates to the field of audio and picture intelligent synchronization, and particularly discloses a video conference audio and picture synchronization method and system, and the method comprises the steps: extracting respective deep semantic features of an audio sample and a video stream after receiving and preliminary processing (such as out-of-order rearrangement, coding and decoding), such as the semantic features of a Mel spectrogram of an audio and the semantic features of a video clip; and carrying out matching search among the high-dimensional semantic features by utilizing a graph learning algorithm. According to the method, the perception of the human eyes and the human ears on the relevance of the sound and picture contents can be simulated, so that the real time offset between the human eyes and the human ears can be more accurately captured and quantified, and the sound and picture playing offset can be accurately estimated even in a complex scene that a traditional timestamp is possibly invalid or inaccurate. And finally, the audio and video data blocks are adjusted and rendered based on the offset, so that the accuracy and robustness of audio and picture synchronization are improved fundamentally.

Description

Technical Field

[0001] The present application relates to the field of intelligent synchronization of audio and video, and more specifically, to a method and system for synchronizing audio and video in video conferencing. Background Art

[0002] With the rapid development of network technology and the growing demand for remote collaboration, video conferencing systems have been widely adopted in various fields, including business, education, and healthcare, becoming an indispensable part of modern communications. During video conferencing, the quality of audio and video synchronization directly impacts user communication efficiency and overall experience. Any perceptible audio and video asynchrony, such as a mismatch between lip movements and voice, can hinder information transmission, distract participants, and even reduce meeting efficiency. The inherent uncertainties of network transmission, such as jitter, packet loss, and the different paths and processing delays that audio and video streams may experience in the network, are the main causes of audio and video asynchrony. Therefore, building an accurate and efficient audio and video synchronization solution for video conferencing is crucial to ensuring high-quality remote communication.

[0003] Existing video conferencing audio and video synchronization solutions typically rely on aligning the timestamp information carried in audio and video packets, such as using the timestamps in the Real-time Transport Protocol (RTP). The sender adds the corresponding timestamps when packaging the audio and video data, and the receiver uses these timestamps for buffering and playback control, attempting to restore the original timing relationship during playback. However, traditional methods that rely solely on time point comparisons to monitor and adjust audio and video synchronization have inherent flaws. For example, when network conditions are complex and variable, or when latency introduced by audio and video encoding, decoding, and rendering processing chains fluctuates unexpectedly, relying solely on raw timestamp comparisons may not accurately reflect the actual perceptual audio and video synchronization status. Sometimes, the timestamps themselves may even contain errors at the source or be improperly modified during transmission, resulting in ineffective timestamp-based synchronization mechanisms and difficulty in addressing subtle desynchronization issues in all scenarios, thus affecting the user experience.

[0004] Therefore, an optimized audio and video synchronization solution for video conferencing is desired. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, the present application is proposed. The embodiments of the present application provide a method and system for synchronizing audio and video in video conferencing. The method extracts the respective deep semantic features of the received and preliminarily processed (such as shuffling and encoding and decoding) audio samples and video streams, such as the semantic features of the mel-spectrogram of the audio and the semantic features of the video clips, and uses a graph learning algorithm to perform a matching search between these high-dimensional semantic features. This method can simulate the perception of the human eye and ear on the relevance of audio and video content, thereby more accurately capturing and quantifying the real time offset between the two. Even in complex scenarios where traditional timestamps may be invalid or inaccurate, the audio and video playback offset can be accurately estimated. Finally, the audio and video data blocks are adjusted and rendered based on this offset, aiming to fundamentally improve the accuracy and robustness of audio and video synchronization, provide users with a more natural and smooth video conferencing experience, and effectively solve the problem of poor synchronization caused by traditional methods relying only on surface time information.

[0006] According to one aspect of the present application, a method for synchronizing audio and video in a video conference is provided, comprising:

[0007] receiving raw video conferencing network data packets from a network interface;

[0008] Identify and separate an audio RTP data packet stream and a video RTP data packet stream from the original video conferencing network data packet;

[0009] The audio RTP data packet stream is stored in an audio receiving buffer, and the video RTP data packet stream is stored in a video receiving buffer, and the audio RTP data packet stream and the video RTP data packet stream are rearranged out of order based on an RTP sequence number to obtain an ordered audio RTP data packet stream and an ordered video RTP data packet stream;

[0010] Extracting an audio sample containing a timestamp and a video stream containing a timestamp from the ordered audio RTP data packet stream and the ordered video RTP data packet stream;

[0011] Determining an audio and video playback offset based on a comparison between the audio sample containing the timestamp and the video stream containing the timestamp;

[0012] Based on the audio and video playback offset, generating a synchronously adjusted audio data block and a video data block;

[0013] The synchronously adjusted audio data blocks and video data blocks are rendered, and the rendered video conference audio and video are displayed on the display screen.

[0014] According to another aspect of the present application, a video conferencing audio and video synchronization system is provided, comprising:

[0015] The conference network data packet acquisition module always receives the original video conference network data packets from the network interface;

[0016] An RTP data packet stream separation module is used to identify and separate an audio RTP data packet stream and a video RTP data packet stream from the original video conferencing network data packet;

[0017] a data packet stream ordered arrangement module, configured to store the audio RTP data packet stream into an audio receiving buffer, store the video RTP data packet stream into a video receiving buffer, and perform random reordering on the audio RTP data packet stream and the video RTP data packet stream based on an RTP sequence number to obtain an ordered audio RTP data packet stream and an ordered video RTP data packet stream;

[0018] A timestamp information extraction module, configured to extract an audio sample containing a timestamp and a video stream containing a timestamp from the ordered audio RTP data packet stream and the ordered video RTP data packet stream;

[0019] an audio and video playback offset detection module, configured to determine an audio and video playback offset based on a comparison between the audio sample containing the timestamp and the video stream containing the timestamp;

[0020] an audio and video synchronization adjustment module, configured to generate an audio data block and a video data block that have undergone synchronization adjustment based on the audio and video playback offset;

[0021] The conference audio and video rendering module always renders the synchronously adjusted audio data blocks and video data blocks, and displays the rendered video conference audio and video on the display screen.

[0022] Compared with the existing technology, the present application provides a method and system for synchronizing audio and video in video conferencing. It extracts the deep semantic features of the received and preliminarily processed (such as shuffling and encoding and decoding) audio samples and video streams, such as the semantic features of the mel-spectrogram of the audio and the semantic features of the video clips, and uses a graph learning algorithm to perform matching searches between these high-dimensional semantic features. This method can simulate the human eye and ear's perception of the relevance of audio and video content, thereby more accurately capturing and quantifying the true time offset between the two. Even in complex scenarios where traditional timestamps may fail or be inaccurate, the audio and video playback offset can be accurately estimated. Ultimately, the audio and video data blocks are adjusted and rendered based on this offset, aiming to fundamentally improve the accuracy and robustness of audio and video synchronization, provide users with a more natural and smooth video conferencing experience, and effectively solve the problem of poor synchronization caused by traditional methods relying solely on surface time information. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 Flowchart of a method for synchronizing audio and video in a video conference according to an embodiment of the present application;

[0025] Figure 2 Schematic diagram of data flow of a method for synchronizing audio and video in a video conference according to an embodiment of the present application;

[0026] Figure 3 A flowchart of determining an audio and video playback offset based on a comparison between the audio sample containing a timestamp and the video stream containing a timestamp according to a method for synchronizing audio and video in a video conference according to an embodiment of the present application;

[0027] Figure 4 A flowchart of a method for synchronizing audio and video in a video conference according to an embodiment of the present application for passing a sequence of the audio mel-spectrogram semantic feature vector and the video stream segment semantic feature vector through a graph-based learning video segment semantic feature search module to obtain a video stream segment-audio semantic search response encoding vector;

[0028] Figure 5 4 is a block diagram of a video conferencing audio and video synchronization system according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0030] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0031] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are illustrative only, and different aspects of the system and method can use different modules.

[0032] Flowcharts are used in this application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously, as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0033] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0034] To address the shortcomings of existing synchronization technologies, which primarily rely on RTP timestamps, a technique that is sensitive to network fluctuations and lacks the ability to accurately reflect the perceived synchronization state, this technology focuses on building an intelligent synchronization mechanism based on the semantic relevance of audio and video content. This mechanism goes beyond discrete timestamps and instead deeply analyzes the intrinsic content of the audio and video streams. This mechanism extracts deep semantic features from received and initially processed (e.g., shuffling and encoding / decoding) audio and video samples—for example, the semantic features of the audio's mel-spectrogram and the semantic features of the video clips—and then uses advanced algorithms such as graph learning to perform a match search between these high-dimensional semantic features. This approach simulates the human eye and ear's perception of the relevance of audio and video content, more accurately capturing and quantifying the true time offset between them. This allows for precise estimation of the audio and video playback offset, even in complex scenarios where traditional timestamps may be invalid or inaccurate. Finally, based on this offset, audio and video data blocks are adjusted and rendered, aiming to fundamentally improve the accuracy and robustness of audio and video synchronization, providing users with a more natural and smooth video conferencing experience. This effectively addresses the poor synchronization performance of traditional methods that rely solely on surface-level time information.

[0035] In the technical solution of the present application, a method for synchronizing audio and video in a video conference is proposed. Figure 1 The present invention is a flowchart of a method for synchronizing audio and video in a video conference according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the method for synchronizing audio and video in a video conference according to an embodiment of the present application. Figure 1 and Figure 2As shown, the video conference audio and video synchronization method according to the embodiment of the present application includes the following steps: S100, receiving the original video conference network data packet from the network interface; S200, identifying and separating the audio RTP data packet stream and the video RTP data packet stream from the original video conference network data packet; S300, storing the audio RTP data packet stream into the audio receiving buffer, and storing the video RTP data packet stream into the video receiving buffer, and re-arranging the audio RTP data packet stream and the video RTP data packet stream based on the RTP sequence number to obtain the ordered audio RTP data packet stream and the ordered S400, extracting the audio samples containing timestamps and the video stream containing timestamps from the ordered audio RTP data packet stream and the ordered video RTP data packet stream; S500, determining the audio and video playback offset based on the comparison between the audio samples containing timestamps and the video stream containing timestamps; S600, generating the audio data blocks and video data blocks after synchronization adjustment based on the audio and video playback offset; S700, rendering the audio data blocks and video data blocks after synchronization adjustment, and displaying the rendered video conference audio and video on the display screen.

[0036] Specifically, in step S100, an original video conferencing network data packet is received from a network interface. It should be understood that since the original video conferencing network data packet completely retains all information of the audio and video data at the network transmission layer, including key metadata such as the protocol identifier used to distinguish audio and video streams, the RTP sequence number to ensure the order of the data packets, and the RTP timestamp for preliminary time alignment, this information is crucial for subsequent audio and video separation, reordering, timestamp extraction, and ultimately accurate synchronization. If it is not based on the original data packet, this information may be lost or changed during transmission or when processed by upper-layer applications, thereby affecting the accuracy of synchronization.

[0037] More specifically, in a specific example of the present application, first, the program initializes the network subsystem and selects a suitable network interface (such as an Ethernet card or a wireless network card), which is the physical or logical channel for the video conferencing data to flow in. Secondly, a network socket is created as the endpoint for the application to communicate with the network protocol stack; for scenarios where RTP is usually carried on UDP, a UDP socket is created and bound to the specific IP address and port number agreed upon by the video conferencing client or server to receive the media stream. If lower-level access is required, such as in scenarios where it is necessary to process IP layer or link layer header information, raw sockets may be used or packet capture libraries such as libpcap / WinPcap may be utilized. These libraries can capture data frames directly from the network adapter level, providing more complete access to network data packets. Next, the socket is placed in a listening state, and the application enters a receive loop, in which it calls receive functions (such as recvfrom for UDP sockets or reading data from the pcap library) in a blocking wait or non-blocking polling of the network interface. Once data arrives, the operating system's network stack pushes packets that meet the binding criteria to the application. Finally, the received data is raw network packets in the form of a byte stream. These packets, unparsed by upper-layer application protocols, contain complete information from the link layer (such as the Ethernet frame header), the network layer (such as the IP header), the transport layer (such as the UDP header), and the application layer (the RTP header and payload). These raw packets are then fed into subsequent processing modules for protocol parsing, audio and video stream separation, and other operations. From a system architecture perspective, this step can be viewed as a dedicated network reception and raw packet storage module, providing unmodified input data for subsequent, sophisticated processing within the entire audio and video synchronization system.

[0038] Specifically, in step S200, the audio RTP data packet stream and the video RTP data packet stream are identified and separated from the original video conferencing network data packet. It should be understood that audio and video, as two different media types, have different data characteristics, encoding and decoding requirements, sensitivity to network delay and jitter, and rendering mechanisms. If separation is not performed, not only will it be impossible to optimize the respective processing flows in a targeted manner, but it will also be impossible to independently extract the respective timestamp information and perform content semantic analysis to implement advanced synchronization strategies. Therefore, the audio and video data mixed in the original network stream must be distinguished based on their inherent identification to form logically independent audio and video data paths.

[0039] Specifically, in one example of this application, the first step, protocol stack parsing, involves performing a bottom-up protocol parsing of the received raw network data packet. This means the program sequentially strips off the link layer header (such as the Ethernet frame header, if captured at the link layer), the network layer header (typically the IP header), and the transport layer header (for RTP, typically the UDP header). Parsing the IP header confirms the source and destination addresses of the packet, while parsing the UDP header reveals the source and destination port numbers. In video conferencing applications, UDP packets that may carry RTP data can be initially screened using a preset port number or a media port number obtained through signaling negotiation (such as an SDP description). The second step involves RTP packet identification and header parsing. After confirming that a UDP packet may carry an RTP payload, the program further parses the UDP payload, i.e., the RTP packet. The RTP header typically has a fixed minimum length (12 bytes) and includes a version number, padding bits, extension bits, CSRC count, marker bit, sequence number, timestamp, and synchronization source identifier (SSRC). The most critical is the "Payload Type" (PT) field.

[0040] Step 3: Stream separation based on payload type: The "Payload Type" field in the RTP header is the key to distinguishing audio from video RTP packets. At the beginning of a session, the communicating parties negotiate the specific Payload Type values ​​corresponding to different media types (e.g., H.264 video, Opus audio) through a signaling protocol (such as SIP with SDP or H.323). For example, one Payload Type value might be agreed upon to represent Opus audio, while another might represent H.264 video. After parsing each RTP packet's Payload Type, the system compares it with a negotiated mapping table. If the Payload Type matches an audio codec, the RTP packet is identified as an audio RTP packet; if it matches a video codec, it is identified as a video RTP packet. Step 4: Stream aggregation and output: Once the media type of an RTP packet is identified, the system directs the packet to the appropriate processing branch. Audio RTP packets are aggregated into an audio RTP packet stream and sent to the audio receive buffer; video RTP packets are aggregated into a video RTP packet stream and sent to the video receive buffer. The Synchronization Source Identifier (SSRC) plays a key role in this process, distinguishing different media streams from the same participant (although more commonly, they are sent using different ports or RTP sessions). Through these steps, the system effectively separates pure audio and video RTP data streams from the mixed raw network data, paving the way for subsequent reordering, decoding, timestamp extraction, and synchronization adjustments. This process effectively implements an RTP stream demultiplexer on the receiving end.

[0041] Specifically, in step S300, the audio RTP data packet stream is stored in an audio receiving buffer, and the video RTP data packet stream is stored in a video receiving buffer, and the audio RTP data packet stream and the video RTP data packet stream are rearranged based on the RTP sequence number to obtain an ordered audio RTP data packet stream and an ordered video RTP data packet stream. It should be understood that due to the inherent unreliability and disorder of packet switching networks such as the Internet that carry video conferencing data, especially when using protocols such as RTP over UDP for real-time transmission. Audio and video data are packaged into independent RTP data packet sequences at the sending end and sent. Due to network path differences, congestion and other transmission anomalies, these data packets may not arrive at the receiving end in the original sending order (i.e., out of order), and their arrival time intervals may also be uneven (i.e., jitter occurs), and even some data packets may be lost. Storing these data packets in dedicated audio and video receive buffers, respectively, firstly decouples the network reception process from the decoding and rendering process, allowing each to operate at its own pace. Secondly, the buffers provide the necessary temporary storage space to absorb network jitter and wait for late packets. Thirdly, and this is the core purpose of this step, it creates the conditions for performing out-of-order reordering. Furthermore, the audio RTP packet stream and the video RTP packet stream are reordered based on the RTP sequence number. Without reordering, audio playback will be unclear due to garbled or missing data, and video playback will experience tearing, stuttering, or jumps. This is because most audio and video codecs, especially those that rely on inter-frame prediction, require a continuous and ordered data stream for correct decoding. The sequence number in the RTP header (a field that increments for each RTP packet sent) provides a clear mechanism for the receiver to detect out-of-order packets and restore the original order.

[0042] More specifically, in one specific example of this application, for each incoming RTP packet (identified as audio or video and directed to the respective buffer), the sequence number is first extracted from its RTP header. Next, the packet is inserted into its designated buffer (audio or video buffer). Such buffers are typically implemented as jitter buffers, which are more complex than simple first-in-first-out queues. Common data structures include lists sorted by sequence number, minimum-priority queues (keyed by sequence number), or circular arrays indexed by sequence number. The choice of buffer type requires a balance between insertion / retrieval efficiency and memory usage. The buffer size should be sufficient to accommodate expected network jitter and reordering depth, but not so large as to introduce excessive end-to-end latency. Finally, a reordering logic continuously monitors the buffer status. This logic maintains a pointer to the next outgoing packet, or expected sequence number. It checks whether a packet with this expected sequence number has arrived and is present in the buffer. If so, the packet is dequeued from the buffer and passed to the next stage (e.g., a decoder), with the expected sequence number incremented. If a packet with the expected sequence number has not arrived, the system waits for a configurable, often adaptive, period of time (i.e., the jitter buffer delay tolerance). This waiting period allows delayed or out-of-order packets to arrive and be correctly placed. If a packet arrives within this window, it is inserted into the buffer, and the reordering logic continues. If the packet still has not arrived after the timeout, it is considered lost. The system may then choose to discard the missing packet's slot and process the next sequence number, or trigger packet loss compensation mechanisms in subsequent decoding stages. Duplicate packets (identified by sequence number) are typically discarded upon arrival or upon detection in the buffer. This continuous cycle of receiving, buffering, reordering, and dequeuing ensures that both the audio and video decoders receive a continuous and correctly sequenced stream of RTP packets, thereby forming the "sequenced audio RTP packet stream" and "sequenced video RTP packet stream" necessary for precise audio and video synchronization.

[0043] Specifically, in step S400, audio samples containing timestamps and a video stream containing timestamps are extracted from the ordered audio RTP packet stream and the ordered video RTP packet stream. It should be understood that the original ordered RTP packet stream encapsulates compressed and encoded audio and video data, which is not directly suitable for subsequent advanced content analysis (such as semantic feature extraction). To enable in-depth analysis of the actual audio and video content and semantic comparison, they must first be restored to a perceptible original or near-original media format. More specifically, in this embodiment of the present application, extracting audio samples containing timestamps and a video stream containing timestamps from the ordered audio RTP packet stream and the ordered video RTP packet stream includes: performing audio encoding and decoding on the ordered audio RTP packet stream to obtain audio samples containing timestamps; and performing video encoding and decoding on the ordered video RTP packet stream to obtain a video stream containing timestamps. This allows the compressed audio and video streams to be converted into a video stream, such as PCM (Pulse Code Modulation) audio samples and a series of original image frames (e.g., in YUV or RGB format). In addition, traditional synchronization monitoring based on time point comparison has defects, and this technical solution aims to determine the offset by comparing the semantic features of the audio and video content itself. Therefore, the audio samples containing timestamps and the video stream containing timestamps are extracted from the ordered audio RTP data packet stream and the ordered video RTP data packet stream by encoding and decoding, with the aim of obtaining an audio sample sequence and a video frame sequence that are closely related to the original RTP timestamp and can represent the actual playback content. These decoded samples and frames not only contain rich content information for analysis, but the timestamps attached to them (usually the playback timestamps generated by the decoder based on the RTP timestamp and the decoding process) also provide an accurate time reference for the subsequent calculation of the offset. Simply put, this step is to convert the data packets at the network transmission level into audio and video materials at the application perception level with precise time stamps that can be used for deep content analysis, preparing for the subsequent audio mel-spectrogram conversion, video segmentation and extraction of their respective semantic features.

[0044] Specifically, in step S500, the audio and video playback offset is determined based on the comparison between the audio sample containing the timestamp and the video stream containing the timestamp. It should be understood that the existing video conferencing audio and video synchronization solution, which only relies on surface time information such as RTP timestamps for synchronization monitoring, is often difficult to accurately reflect the audio and video synchronization status at the user's actual perception level when faced with complex network environments and variable processing delays, resulting in poor synchronization effect. Therefore, in order to overcome the limitations of traditional methods, in the technical solution of the present application, the audio and video playback offset is further determined based on the comparison between the audio sample containing the timestamp and the video stream containing the timestamp. By analyzing and comparing the intrinsic connection between the audio and video content itself, the actual deviation in the playback timing of the two, that is, the audio and video playback offset, is accurately calculated. This is different from simply comparing the timestamps at the time of packaging, but goes deep into the semantic information level carried by the decoded audio samples and video frames. For example, by converting audio samples into audio mel-spectrograms and extracting their semantic feature vectors, the video stream is segmented and a sequence of semantic feature vectors of the video segments is extracted. Then, using the video segment semantic feature search algorithm learned in the figure, these semantic features are matched and aligned. This content-based comparison aims to find the time point at which the audio and video content are most semantically matched, thereby determining whether there is a lead or lag between them, as well as the specific offset duration. The ultimate goal is to obtain a highly reliable audio and video playback offset estimate decoded value that can guide the subsequent synchronization adjustment module to accurately compensate or adjust the playback timing.

[0045] Figure 3 The flowchart of determining the audio and video playback offset based on the comparison between the audio sample containing the timestamp and the video stream containing the timestamp according to the video conference audio and video synchronization method of the embodiment of the present application is as follows. Figure 3 As shown, according to the video conferencing audio and video synchronization method of an embodiment of the present application, step S500 includes: S510, after converting the audio sample containing the timestamp into an audio mel-spectrogram, extracting the mel-spectrogram features in the audio mel-spectrogram to obtain an audio mel-spectrogram semantic feature vector; S520, performing video stream segment semantic feature extraction on the video stream containing the timestamp to obtain a sequence of video stream segment semantic feature vectors; S530, passing the audio mel-spectrogram semantic feature vector and the sequence of video stream segment semantic feature vectors through a graph learning-based video segment semantic feature search module to obtain a video stream segment-audio semantic search response coding vector; S540, determining an estimated decoding value of the audio and video playback offset based on the video stream segment-audio semantic search response coding vector.

[0046] Specifically, in step S510, after the audio sample containing the timestamp is converted into an audio mel-spectrogram, mel-spectrogram features are extracted from the audio mel-spectrogram to obtain an audio mel-spectrogram semantic feature vector. It should be understood that although the original, decoded audio sample (such as PCM data) contains complete audio information, its data dimension is high and redundancy is large, and it is very difficult to directly extract features from it that can represent high-level semantic information such as speech content, speaker emotion, or specific sound events. In order to achieve content-based audio and video synchronization, the system requires a more compact, discriminative, and closer to human auditory perception representation of audio content. The mel-spectrogram is such an ideal intermediate representation. It converts the audio signal to a mel-frequency scale by simulating the human ear's perception of sounds of different frequencies, thereby more effectively highlighting sound features related to human perception. Therefore, in the technical solution of the present application, after the audio sample containing the timestamp is converted into an audio mel-spectrogram, mel-spectrogram features are extracted from the audio mel-spectrogram to obtain an audio mel-spectrogram semantic feature vector.

[0047] More specifically, in an embodiment of the present application, after converting the audio sample containing the timestamp into an audio mel-spectrogram, extracting mel-spectrogram features from the audio mel-spectrogram to obtain an audio mel-spectrogram semantic feature vector includes: after converting the audio sample containing the timestamp into an audio mel-spectrogram, passing the audio mel-spectrogram through a SoundNet-based mel-spectrogram feature extractor to obtain the audio mel-spectrogram semantic feature vector. In other words, converting to an audio mel-spectrogram maps the time-domain audio signal into a graphical time-frequency representation, which reveals the distribution of sound energy over time and frequency (Mel scale) and is inherently easier to analyze the composition and changes of the sound than the original waveform. Secondly, and more crucially, a deep neural network model, such as a SoundNet-based mel-spectrogram feature extractor, is used to further extract an "audio mel-spectrogram semantic feature vector" from the mel-spectrogram. Models such as SoundNet, trained on large-scale audio datasets, can learn complex mappings from acoustic features to high-level semantic concepts (such as speech, music, and ambient sound). Therefore, the goal is to generate a low-dimensional, dense vector representation that captures the core semantic information of the audio content. This audio mel-spectrogram semantic feature vector captures the "meaning" of the audio clip, not just its physical properties. This semantic feature vector serves as the basis for subsequent comparison and matching with the semantic features of the video content, enabling the system to "understand" what the audio is saying or what is happening. This allows subsequent audio-visual synchronization determination to no longer be limited to simple timestamps, but to compare the "content" of the audio with the "visual content" presented in the video in a higher-dimensional semantic space. For example, it can help determine whether the acoustic feature vector of the word "hello" heard is temporally aligned with the visual feature vector of the person's mouth opening to say "hello" in the video. This feature, based on deep semantic understanding, provides high-quality, information-rich input for subsequent precise matching and offset estimation between audio and video clips using advanced techniques such as graph learning, significantly improving the accuracy and robustness of audio-visual synchronization detection, ultimately enhancing the user experience.

[0048] Specifically, in step S520, the video stream containing the timestamp is subjected to video stream segment semantic feature extraction to obtain a sequence of video stream segment semantic feature vectors. It should be understood that the original decoded video stream is composed of a series of continuous image frames. Directly comparing the original pixel data not only requires a huge amount of calculation, but also makes it difficult to capture the dynamic semantic information in the video content that changes over time, such as the character's movements, lip changes or scene transitions, etc., and these dynamic semantics are precisely the key clues to determine whether the audio and video are synchronized. In addition, simple static frame analysis or comparison based on simple features cannot effectively cope with the diversity and complexity of video content, nor can it be well matched with the semantic information extracted from the audio content. Therefore, in the technical solution of the present application, video stream segment semantic feature extraction is further performed on the video stream containing the timestamp to obtain a sequence of video stream segment semantic feature vectors.

[0049] More specifically, in an embodiment of the present application, video stream segment semantic feature extraction is performed on the video stream containing timestamps to obtain a sequence of video stream segment semantic feature vectors, including: segmenting the video stream containing timestamps to obtain a sequence of video stream segments; and passing the sequence of video stream segments through a video segment semantic feature extractor based on a three-dimensional convolutional neural network model to obtain a sequence of video stream segment semantic feature vectors. That is, first, the continuous video stream is decomposed into units of a certain duration that may be relatively semantically complete. This is done to reduce processing complexity, but more importantly, many meaningful visual events (such as the changes in lip shape corresponding to the pronunciation of a word) have time spans, and analyzing segments can better capture their dynamic characteristics than analyzing single frames. Next, each video segment is mapped from a high-dimensional pixel space to a low-dimensional feature vector space that can represent its core spatiotemporal semantic information through a video segment semantic feature extractor based on a three-dimensional convolutional neural network model. 3D convolutional neural networks are particularly well-suited for processing temporal data such as video. They can simultaneously learn spatial features within video frames (such as object shape and texture) and temporal relationships between frames (such as motion patterns and trends), thereby extracting features rich in dynamic semantics. These feature vectors are designed to capture "what happens" in a video clip, not just "what the image looks like," providing high-quality representation of video content for subsequent precise alignment with audio semantic features.

[0050] Specifically, in step S530, the sequence of the audio mel-spectrogram semantic feature vector and the video stream segment semantic feature vector is passed through a graph-learning-based video segment semantic feature search module to obtain a video stream segment-audio semantic search response encoding vector. It should be understood that traditional audio and video synchronization relies on timestamp alignment, but complex factors such as network jitter and encoding and decoding delay often cause distortion of time information and cannot reflect the true perceptual synchronization state. As heterogeneous data streams, audio and video are the essential basis for the human eye and ear to judge synchronization through their deep semantic associations (such as the correspondence between lip movements and speech content). However, discrete semantic feature vectors are difficult to directly quantify timing offsets, and structured modeling is required to capture the dynamic correlation of audio and video content in the semantic space. Therefore, in the technical solution of the present application, the sequence of the audio mel-spectrogram semantic feature vector and the video stream segment semantic feature vector is further passed through a graph-learning-based video segment semantic feature search module to obtain a video stream segment-audio semantic search response encoding vector. The graph-learning-based video segment semantic feature search module organizes the semantic features of a sequence of semantic feature vectors of video stream segments into graph-structured nodes, explicitly modeling the spatiotemporal relationships between segments (such as the continuity of character movements). Using graph learning algorithms, it dynamically "retrieves" the best matching locations for audio semantic features within this structured semantic network. This simulates the human perceptual system's understanding of the relationship between sound and image, locating the optimal synchronization point between audio and video streams at the content level (rather than the temporal level), providing a semantic-level decision-making basis for offset estimation.

[0051] Figure 4 The flowchart of the method for synchronizing audio and video in a video conference according to an embodiment of the present application is to pass the sequence of the audio mel-spectrogram semantic feature vector and the video stream segment semantic feature vector through a video segment semantic feature search module based on graph learning to obtain a video stream segment-audio semantic search response encoding vector. Figure 4As shown, according to the video conferencing audio and video synchronization method of the embodiment of the present application, step S530 includes: S531, performing information condensation on each video stream segment semantic feature vector in the sequence of the video stream segment semantic feature vectors to obtain a set of video stream segment semantic condensation coding vectors; S532, constructing a sparse association topology matrix between video stream segment semantic feature nodes based on the semantic association between any two video stream segment semantic condensation coding vectors in the set of video stream segment semantic condensation coding vectors; S533, inputting the set of video stream segment semantic condensation coding vectors and the sparse association topology matrix between video stream segment semantic feature nodes into a graph convolutional network model-based imitation graph construction engine to obtain a video stream segment semantic feature imitation graph coding matrix; S534, performing information query in the video stream segment semantic feature imitation graph coding matrix based on the audio Mel-spectrogram semantic feature vector to obtain the video stream segment-audio semantic search response coding vector.

[0052] More specifically, in step S531, information is condensed on each video stream segment semantic feature vector in the sequence of video stream segment semantic feature vectors to obtain a set of video stream segment semantic condensed coding vectors, which is expressed as:

[0053]

[0054]

[0055]

[0056] in, is a sequence of semantic feature vectors of the video stream segments, are the first, second, and third semantic feature vectors in the sequence of the video stream segment. and Semantic feature vectors of video stream segments, and are the trainable weight matrix and the trainable bias vector, respectively. for activation function, for The norm of is a set of semantically condensed encoding vectors for video stream segments, are the first, second, and third vectors in the set of semantic condensed coding vectors of video stream segments. and Semantically condensed encoding vectors of video stream segments.

[0057] It's understandable that while the original semantic feature vector sequences of video stream clips contain rich spatiotemporal information, they suffer from high-dimensional redundancy, noise, and a mixture of non-salient semantic features. For example, in video conferencing scenarios, irrelevant information such as background noise, brief image jitter, or non-lip movements (such as hand gestures) can dilute core semantics (e.g., lip movements). Directly using the original feature sequences for graph learning results in computational inefficiency and distorted association modeling. Therefore, through feature refinement and information concentration mechanisms, redundant noise is removed and key semantic units are strengthened. Essentially, this involves semantically purifying the high-dimensional feature space—preserving dynamic visual patterns strongly associated with the audio content (e.g., lip opening and closing, facial muscle changes corresponding to specific syllables) while compressing repetitive or low-value information in the temporal dimension (e.g., static background). The concentration process recalibrates the feature space through nonlinear transformations (e.g., norm weighting and ReLU activation in the formula), making each concentrated vector a "visual atom" representing the core semantics of the video clip, laying the foundation for the subsequent construction of an accurate semantic association graph.

[0058] Accordingly, according to an embodiment of the present application, step S532, based on the semantic correlation between any two semantic condensed coding vectors of the video stream fragments in the set of semantic condensed coding vectors of the video stream fragments, constructs a sparse association topology matrix between semantic feature nodes of the video stream fragments, including: calculating the semantic correlation between any two semantic condensed coding vectors of the video stream fragments in the set of semantic condensed coding vectors of the video stream fragments to obtain the association topology matrix between semantic feature nodes of the video stream fragments; inputting the association topology matrix between semantic feature nodes of the video stream fragments into a gated mask network to obtain the sparse association topology matrix between semantic feature nodes of the video stream fragments.

[0059] More specifically, the semantic association between any two semantic condensed coding vectors of the video stream segments in the set of semantic condensed coding vectors of the video stream segments is calculated to obtain a topological matrix of association between semantic feature nodes of the video stream segments, which is expressed as follows:

[0060]

[0061] in, is vector concatenation, and are the semantic association weight matrix and the semantic association bias vector respectively, for and Semantic association vectors between video stream segments are condensed. The length of the semantic association vector for the semantic concentration of the video stream segment, The eigenvalue position of the semantic association vector for the semantic concentration of the video stream segment, is the association topology matrix between semantic feature nodes of video stream segments. and The semantic relevance between them.

[0062] It's understandable that while the features of video clips after information condensation have been refined to reveal core semantic units (such as lip movements, gestures, and other key visual elements), the isolated existence of these units on the timeline cannot reflect the human perception of continuity in dynamic images. Visual semantics such as a speaker's lip movements and facial expressions in video conferencing have strong temporal dependencies (for example, the lip transition from the syllable "o" to "u" requires coherent modeling), and traditional linear processing would disrupt this dynamic evolutionary logic. Furthermore, network packet loss or rendering jitter can cause discontinuous arrival of video clips, necessitating the establishment of a semantic association network that transcends temporal order. The goal of calculating semantic associations and constructing a topological matrix of associations between semantic feature nodes in video stream segments is to construct a dynamic relationship graph of visual semantics. By quantifying the semantic affinity between any two condensed encoding vectors (for example, calculating the dynamic similarity of lip movement patterns or the emotional consistency of facial expressions), discrete video clips are mapped into nodes and edges in a graph structure. This process is essentially to reconstruct the implicit causal chain of visual events in a high-dimensional semantic space. For example, related lip segments in the pronunciation process (even if they are discontinuous on the timeline due to network jitter) are connected through high-correlation edges to form a topological network that conforms to the mechanism of speech production, providing structured priors for subsequent graph learning.

[0063] More specifically, the association topology matrix between the semantic feature nodes of the video stream segment is input into the gated mask network to obtain the sparse association topology matrix between the semantic feature nodes of the video stream segment, which is expressed as follows:

[0064] ;

[0065] in, is the mask weight vector, is the mask bias weight parameter, is the correlation topology matrix between semantic feature nodes of video stream segments, for function, is the sparse correlation topology matrix between semantic feature nodes of video stream segments. and The masked semantic correlation between .

[0066] It's understandable that while the initial topology of semantic feature nodes in video stream segments quantifies the semantic similarity between segments, a large number of non-causal connections in video conferencing scenarios (such as static frames with similar backgrounds and repetitive movements unrelated to pronunciation) can create noisy connections, leading to an overly dense graph structure. For example, during a conversation between two people, a listener's head nods may coincidentally correlate with the speaker's audio. If such spurious correlations are not suppressed, they can severely disrupt the core lip-to-speech matching. Therefore, a gated mask network (GMA) can achieve intelligent sparsification and semantic refinement of the graph structure. The GMA acts as an adaptive filter, dynamically evaluating the semantic necessity of each edge through learnable nonlinear transformations (such as sigmoid gating). It retains visual dynamic connections strongly related to pronunciation (such as high-weight edges between consecutive segments of lip changes) while weakening or severing incidental connections (such as low-weight edges between gestures and irrelevant environmental elements). This process essentially models the attention mechanism of human audiovisual perception as a graph topology optimization—simulating the human brain's ability to automatically focus on the core semantic chain (lip movement flow) while ignoring background interference when processing audio-visual synchronization.

[0067] More specifically, in step S533, the set of semantic condensed coding vectors of the video stream segments and the sparse correlation topology matrix between the semantic feature nodes of the video stream segments are input into the graph-like construction engine based on the graph convolutional network model to obtain the semantic feature graph-like coding matrix of the video stream segments, which is expressed as follows:

[0068]

[0069] in, is the sparse correlation topology matrix between semantic feature nodes of video stream segments. and The mask semantic association between The semantic features of the video stream segment are encoded in the graph-like matrix and The semantic correlation between the semantic features of video stream segments.

[0070] It should be understood that although the sparse correlation topology matrix between semantic feature nodes in video stream segments has extracted key visual semantic connections (such as the causal chain of lip shape changes), the discrete video stream segment semantic node features and binary edge connections are still unable to model the complex spatiotemporal dynamics of human audiovisual perception. The visual semantics of articulatory movements in video conferencing are inherently continuous and fluid—for example, the dynamic process of lip movement from consonant explosion to vowel extension. Its semantic integrity depends on nonlinear interactions between segments (such as the constraining effect of the preceding lip shape on the subsequent articulatory movement). Traditional feature pooling or simple graph traversal will miss these high-order evolutionary laws. Therefore, through the processing of the graph-like atlas construction engine based on the graph convolutional network model, the multi-layer neighborhood aggregation mechanism of the graph convolutional network (GCN) can iteratively fuse multi-hop neighborhood information into the condensed encoding vector of each video segment on an optimized topological structure. For example, the lip shape features of segment t will absorb the semantic flow of the preparatory lip shape of segment t-1 and the closing lip shape of segment t+1, forming a dynamic "articulatory unit embedding." The graph-like construction engine plays the role of a manifold learner in this process - mapping discrete visual events into a continuous high-dimensional semantic surface, where the geodesic distance between adjacent nodes on the manifold directly corresponds to the physiological coherence of lip shape changes, thereby accurately simulating the human visual system's holistic understanding of dynamic lip movements.

[0071] More specifically, according to an embodiment of the present application, step S534, performing information query in the video stream segment semantic feature imitation graph coding matrix based on the audio mel-spectrogram semantic feature vector to obtain the video stream segment-audio semantic search response coding vector, includes: inputting the audio mel-spectrogram semantic feature vector and the video stream segment semantic feature imitation graph coding matrix into a feature query response engine to obtain the video stream segment-audio semantic search response coding vector, which is expressed as follows:

[0072]

[0073]

[0074] in, is the audio Mel-spectrogram semantic feature vector, are the row vectors in the atlas encoding matrix of the semantic features of the video stream segment, for function, is the feature query factor, is the point product by position, Encode vectors for video stream segment-audio semantic search responses.

[0075] It should be understood that although the semantic feature graph-like encoding matrix of video stream segments constructs a continuous manifold of video semantics (such as the dynamic surface of lip movement), audio features, as independent time slices, must be precisely anchored to this visual semantic surface. In video conferencing scenarios with sudden network jitter or multiple people speaking, the timing of the audio and video streams may shift non-uniformly (for example, a syllable of speech lags behind the corresponding lip frame due to network latency). Traditional linear matching cannot adapt to this dynamic distortion. Through the feature query response engine, the audio Mel-spectrogram semantic feature vector is used as a dynamic query signal through a differentiable attention mechanism to perform a probabilistic walk on the visual semantic manifold generated by the video graph-like encoding matrix: the semantic affinity between the query vector and each video segment node in the graph is calculated (for example, the strong correlation between plosive audio and lip closure frames), and the relevant nodes are weightedly aggregated in a manifold-aware manner based on the attention weights. This process simulates the human brain's judgment mechanism for audio-visual synchronization: when a specific phoneme is heard, it automatically focuses on the moment in the video stream that is most likely to show the corresponding lip shape, even if the frame has shifted on the timeline due to network problems.

[0076] Preferably, although the graph neural network model has captured the high-order relationship and global structural information under the graph structure relative to the semantic condensed coding vector of each video stream segment, due to the sparsity introduced by the gated mask network to the topological structure of the graph, the row vectors in the semantic feature-like graph encoding matrix of the video stream segment are not well-defined. , we still expect it to show the same statistical laws as much as possible at the macro level of high-order distribution outside the graph structure, that is, we expect to achieve each row vector The high-order global correlation of .

[0077] First, use the semicircle distribution model to calculate each row vector Fixed neighborhood mapping for :

[0078] ;

[0079] in, For all row vectors The mean vector is obtained by averaging the corresponding positions of . Fixed neighborhood mapping vector It can capture higher-order statistical dependencies such as supercovariance relationships that go beyond traditional covariance.

[0080] Then, the fixed neighborhood mapping vector is calculated Dynamic aggregation indicators:

[0081]

[0082] in is a vector No. The eigenvalue of the location.

[0083] Finally, the dynamic aggregation index For the row vector Perform weighted optimization for .

[0084] That is, considering that the statistical laws of high-order macroscopic distribution are mainly dominated by high-order statistical moments, in order to make the high-order macroscopic distribution correlation structure In order to ensure consistency between the two, it is necessary to constrain the global dynamic symmetry through the dynamic cumulative index under the fixed neighborhood mapping, so as to ensure the consistent propagation of macro-distribution statistics under the graph structure and avoid statistically irrelevant responses such as cascading failures under the graph structure.

[0085] Specifically, in step S540, an estimated decoded value of the audio and video playback offset is determined based on the video stream segment-audio semantic search response encoding vector. It should be understood that while the video stream segment-audio semantic search response encoding vector captures the positioning information of audio features in the visual semantic manifold (such as the topological correlation strength and direction of lip shape and speech), it is essentially an abstract semantic coordinate in a high-dimensional space (such as a 128-dimensional vector) and cannot directly correspond to the audio and video playback offset. Especially in multi-person conversation scenarios, the video stream segment-audio semantic search response encoding vector may contain both a strong matching signal from the main speaker and weak noise from other participants, necessitating a deterministic mapping relationship from semantic space to temporal space. Therefore, in the technical solution of the present application, an estimated decoded value of the audio and video playback offset is further determined. In particular, in an embodiment of the present application, determining the estimated decoded value of the audio and video playback offset based on the video stream segment-audio semantic search response encoding vector includes: passing the video stream segment-audio semantic search response encoding vector through a decoder-based offset estimator to obtain the estimated decoded value of the audio and video playback offset. During the training phase, the decoder uses massive amounts of labeled data (artificially generated frame-accurate audio / video offset samples) to learn physical laws. For example, the sign of the principal component of the response vector corresponds to an audio advance or lag, and its magnitude is hyperbolic tangent to the actual offset in milliseconds. This process essentially constructs a "semantic-to-temporal" converter, translating logical associations in human audiovisual perception (such as the correspondence between specific phonemes and lip frames) into calculable physical delay values.

[0086] Specifically, in step S600, synchronously adjusted audio and video data blocks are generated based on the audio and video playback offset. It should be understood that since the audio and video playback offset reflects the actual misalignment between the audio and video streams on the playback timeline (e.g., audio lags behind video by 200ms), if it is not actively corrected, it will lead to perceptual anomalies such as mismatch between lip movements and speech, and disconnection between movement and sound. Therefore, in order to eliminate the negative impact of timing deviation on user experience, the technical solution of this application generates synchronously adjusted audio and video data blocks based on the audio and video playback offset.

[0087] More specifically, in one example of this application, the numerical value and direction of the audio / video playback offset are first analyzed (a positive value indicates audio advance, a negative value indicates video advance). A correction strategy is then selected based on the absolute value of the offset: when the offset is less than a threshold (typically ≤80ms), audio time stretching is used for fine-tuning; when the offset exceeds the threshold or the video lags significantly, video frame skipping or interpolation is enabled. This decision is based on the differences in human audiovisual perception: the human ear is sensitive to audio interruptions, while vision can tolerate a small number of frame skips. Dynamic audio stream scaling: For small offsets, the waveform similarity overlay (WSOLA) algorithm is used to perform non-uniform resampling of audio data blocks. If the audio lags, the audio duration is compressed (e.g., removing silence or shortening vowels); if the audio advances, microsecond silence is inserted or consonants are stretched. This process is performed in real time within the audio rendering pipeline, maintaining the fundamental frequency to avoid pitch distortion while ensuring smooth transitions between data block boundaries, achieving imperceptible adjustments within ±100ms. Video stream frame operation: For significant video lag (such as offset > 100ms), the system calculates the number of frames to be skipped (offset / frame interval). A reference frame management system is used: non-key frames (B / P frames) are discarded, the most recent key frame (I frame) is retained as a benchmark, and transition frames are generated based on motion vector interpolation to fill the visual gap. Conversely, if the video is ahead, the previous frame is copied or interpolated frames are generated based on the optical flow method to extend the display time. All operations are performed in the GPU-accelerated video buffer to ensure that there is no tearing in the picture. After completing the single media stream adjustment, the video data block is re-anchored based on the audio timeline: the adjusted video frame timestamp is aligned with the audio data block start timestamp, and the rendering queue is updated through the timestamp mapping table. This process requires dynamic monitoring of the buffer water level to prevent the adjustment from causing buffer underrun or overflow, and triggering the adaptive delay compensation mechanism when necessary.

[0088] Specifically, in step S700, the synchronized audio and video data blocks are rendered, and the rendered video conference audio and video are displayed on the display screen. It should be understood that the synchronized audio and video data blocks are still in an encoded or intermediate representation state (such as PCM audio frames and YUV video frames) and need to be converted into sound waves and optical signals by physical devices. Therefore, the synchronized audio and video data blocks are rendered, and the rendered video conference audio and video are displayed on the display screen. The core technical goal of this process is to achieve precise spatiotemporal reproduction of audio and video signals: under the physical constraints of the display refresh cycle and the speaker sound wave emission, ensure that the adjusted synchronization relationship does not cause secondary distortion when output by the terminal. More specifically, in a specific example of the present application, synchronously adjusted audio data blocks and video data blocks can be input, and then the cross-modal features between the synchronously adjusted audio data blocks and video data blocks can be fused through a spatiotemporal alignment convolution layer (ST-Conv). This cross-modal feature is then input into a large model to generate a PCM stream to drive the speaker and an RGB pixel matrix directly for screen display, thereby generating rendered video conference audio and video and displaying it on the display screen.

[0089] In summary, according to the embodiment of the present application, the method for synchronizing audio and video in video conferencing is explained, which extracts the respective deep semantic features of the audio samples and video streams after receiving and preliminary processing (such as shuffling, encoding and decoding) - such as the semantic features of the mel-spectrogram of the audio and the semantic features of the video clips, and uses a graph learning algorithm to perform matching searches between these high-dimensional semantic features. This method can simulate the perception of the human eye and ear on the relevance of audio and video content, thereby more accurately capturing and quantifying the real time offset between the two, and accurately estimating the audio and video playback offset even in complex scenarios where traditional timestamps may fail or be inaccurate. Ultimately, the audio and video data blocks are adjusted and rendered based on this offset, aiming to fundamentally improve the accuracy and robustness of audio and video synchronization, provide users with a more natural and smooth video conferencing experience, and effectively solve the problem of poor synchronization caused by traditional methods relying only on surface time information.

[0090] Furthermore, a video conferencing audio and video synchronization system is also provided.

[0091] Figure 5 FIG is a block diagram of a video conferencing audio and video synchronization system according to an embodiment of the present application. Figure 5As shown, the video conferencing audio and video synchronization system 500 according to the embodiment of the present application includes: a conference network data packet acquisition module 510, which always receives the original video conferencing network data packet from the network interface; an RTP data packet stream separation module 520, which is used to identify and separate the audio RTP data packet stream and the video RTP data packet stream from the original video conferencing network data packet; a data packet stream ordered arrangement module 530, which is used to store the audio RTP data packet stream into the audio receiving buffer, and store the video RTP data packet stream into the video receiving buffer, and rearrange the audio RTP data packet stream and the video RTP data packet stream in a disordered manner based on the RTP sequence number to obtain an ordered audio RTP data packet stream and an ordered video RTP data packet stream. audio RTP data packet stream; a timestamp information extraction module 540, for extracting audio samples containing timestamps and video streams containing timestamps from the ordered audio RTP data packet stream and the ordered video RTP data packet stream; an audio and video playback offset detection module 550, for determining the audio and video playback offset based on the comparison between the audio samples containing timestamps and the video stream containing timestamps; an audio and video synchronization adjustment module 560, for generating synchronized audio data blocks and video data blocks based on the audio and video playback offset; a conference audio and video rendering module 570, for always rendering the synchronized audio data blocks and video data blocks, and displaying the rendered video conference audio and video on the display screen.

[0092] As described above, the video conferencing audio and video synchronization system 500 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server equipped with a video conferencing audio and video synchronization algorithm. In one possible implementation, the video conferencing audio and video synchronization system 500 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the video conferencing audio and video synchronization system 500 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal. Of course, the video conferencing audio and video synchronization system 500 can also be one of the many hardware modules of the wireless terminal.

[0093] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for synchronizing audio and video in a video conference, characterized in that: include: receiving raw video conferencing network data packets from a network interface; Identify and separate an audio RTP data packet stream and a video RTP data packet stream from the original video conferencing network data packet; The audio RTP data packet stream is stored in an audio receiving buffer, and the video RTP data packet stream is stored in a video receiving buffer, and the audio RTP data packet stream and the video RTP data packet stream are rearranged out of order based on an RTP sequence number to obtain an ordered audio RTP data packet stream and an ordered video RTP data packet stream; Extracting an audio sample containing a timestamp and a video stream containing a timestamp from the ordered audio RTP data packet stream and the ordered video RTP data packet stream; Determining an audio and video playback offset based on a comparison between the audio sample containing the timestamp and the video stream containing the timestamp; Based on the audio and video playback offset, generating a synchronously adjusted audio data block and a video data block; The synchronously adjusted audio data blocks and video data blocks are rendered, and the rendered video conference audio and video are displayed on the display screen.

2. The method for synchronizing audio and video in a video conference according to claim 1, wherein: Extracting an audio sample containing a timestamp and a video stream containing a timestamp from the ordered audio RTP data packet stream and the ordered video RTP data packet stream, comprising: Performing audio encoding and decoding on the ordered audio RTP data packet stream to obtain audio samples including timestamps; Video encoding and decoding are performed on the ordered video RTP data packet stream to obtain a video stream containing a timestamp.

3. The method for synchronizing audio and video in a video conference according to claim 2, wherein: Determining an audio and video playback offset based on a comparison between the audio sample containing the timestamp and the video stream containing the timestamp includes: After converting the audio sample containing the timestamp into an audio mel-spectrogram, extracting mel-spectrogram features from the audio mel-spectrogram to obtain an audio mel-spectrogram semantic feature vector; Performing video stream segment semantic feature extraction on the video stream containing the timestamp to obtain a sequence of video stream segment semantic feature vectors; Passing the sequence of the audio mel-spectrogram semantic feature vector and the video stream segment semantic feature vector through a graph-based video segment semantic feature search module to obtain a video stream segment-audio semantic search response encoding vector; Based on the video stream segment-audio semantic search response coding vector, an estimated decoded value of the audio and video playback offset is determined.

4. The method for synchronizing audio and video in a video conference according to claim 3, wherein: After converting the audio sample containing the timestamp into an audio mel-spectrogram, extracting mel-spectrogram features in the audio mel-spectrogram to obtain an audio mel-spectrogram semantic feature vector, including: after converting the audio sample containing the timestamp into an audio mel-spectrogram, passing the audio mel-spectrogram through a mel-spectrogram feature extractor based on SoundNet to obtain the audio mel-spectrogram semantic feature vector.

5. The method for synchronizing audio and video in a video conference according to claim 4, wherein: Extracting semantic features of video stream segments from the video stream containing the timestamp to obtain a sequence of semantic feature vectors of the video stream segments includes: Segmenting the video stream containing the timestamps into segments to obtain a sequence of video stream segments; The sequence of the video stream segments is passed through a video segment semantic feature extractor based on a three-dimensional convolutional neural network model to obtain a sequence of semantic feature vectors of the video stream segments.

6. The method for synchronizing audio and video in a video conference according to claim 5, wherein: The method further comprises: applying the sequence of the audio mel-spectrogram semantic feature vector and the video stream segment semantic feature vector to a video segment semantic feature search module based on graph learning to obtain a video stream segment-audio semantic search response encoding vector, including: performing information condensation on each video stream segment semantic feature vector in the sequence of video stream segment semantic feature vectors to obtain a set of video stream segment semantic condensed coding vectors; Constructing a sparse correlation topology matrix between semantic feature nodes of video stream segments based on the semantic correlation between any two semantic condensed coding vectors of video stream segments in the set of semantic condensed coding vectors of video stream segments; Inputting the set of semantic condensed coding vectors of the video stream segments and the sparse correlation topology matrix between semantic feature nodes of the video stream segments into a graph-like construction engine based on a graph convolutional network model to obtain a semantic feature-like graph-like coding matrix of the video stream segments; Based on the audio mel-spectrogram semantic feature vector, information query is performed in the video stream segment semantic feature imitation spectrum coding matrix to obtain the video stream segment-audio semantic search response coding vector.

7. The method for synchronizing audio and video in a video conference according to claim 6, wherein: Based on the semantic correlation between any two semantic condensed coding vectors of the video stream segments in the set of semantic condensed coding vectors of the video stream segments, a sparse correlation topology matrix between semantic feature nodes of the video stream segments is constructed, including: Calculating the semantic association between any two video stream segment semantic condensation coding vectors in the set of the video stream segment semantic condensation coding vectors to obtain a topological matrix of association between semantic feature nodes of the video stream segment; The association topology matrix between semantic feature nodes of the video stream segment is input into a gated mask network to obtain a sparse association topology matrix between semantic feature nodes of the video stream segment.

8. The method for synchronizing audio and video in a video conference according to claim 7, wherein: Based on the audio mel-spectrogram semantic feature vector, information query is performed in the video stream segment semantic feature imitation graph coding matrix to obtain the video stream segment-audio semantic search response coding vector, including: inputting the audio mel-spectrogram semantic feature vector and the video stream segment semantic feature imitation graph coding matrix into a feature query response engine to obtain the video stream segment-audio semantic search response coding vector.

9. The method for synchronizing audio and video in a video conference according to claim 8, wherein: Based on the video stream segment-audio semantic search response coding vector, an estimated decoded value of the audio and video playback offset is determined, including: passing the video stream segment-audio semantic search response coding vector through a decoder-based offset estimator to obtain an estimated decoded value of the audio and video playback offset.

10. A video conferencing audio and video synchronization system, characterized in that: include: The conference network data packet acquisition module always receives the original video conference network data packets from the network interface; An RTP data packet stream separation module is used to identify and separate an audio RTP data packet stream and a video RTP data packet stream from the original video conferencing network data packet; a data packet stream ordered arrangement module, configured to store the audio RTP data packet stream into an audio receiving buffer, store the video RTP data packet stream into a video receiving buffer, and perform random reordering on the audio RTP data packet stream and the video RTP data packet stream based on an RTP sequence number to obtain an ordered audio RTP data packet stream and an ordered video RTP data packet stream; A timestamp information extraction module, configured to extract an audio sample containing a timestamp and a video stream containing a timestamp from the ordered audio RTP data packet stream and the ordered video RTP data packet stream; an audio and video playback offset detection module, configured to determine an audio and video playback offset based on a comparison between the audio sample containing the timestamp and the video stream containing the timestamp; an audio and video synchronization adjustment module, configured to generate an audio data block and a video data block that have undergone synchronization adjustment based on the audio and video playback offset; The conference audio and video rendering module always renders the synchronously adjusted audio data blocks and video data blocks, and displays the rendered video conference audio and video on the display screen.

Images