A data synchronization method and apparatus

Abstract

The application provides a data synchronization method and device, and relates to the technical field of data processing. The method comprises the following steps: acquiring first modal data and second modal data; extracting a first modal event from the first modal data and extracting a second modal event from the second modal data; determining a matching relationship between the first modal event and the second modal event; and synchronizing the first modal data and the second modal data based on the matching relationship.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data synchronization method and apparatus. Background Technology

[0002] With the development of multimodal artificial intelligence technology, the fusion of audio and visual information is widely used in scenarios such as video conferencing, intelligent voice assistants, and target speech separation. Therefore, it is necessary to consider how to synchronize information from different modalities, such as audio and vision, for subsequent processing. Summary of the Invention

[0003] This application is made in view of at least one of the above-mentioned technical problems existing in the prior art, and the application can improve synchronization accuracy and robustness.

[0004] In a first aspect, embodiments of this application provide a data synchronization method, including: Acquire first modal data and second modal data; Extract a first modal event from the first modal data, and extract a second modal event from the second modal data; Determine the matching relationship between the first modal event and the second modal event; Based on the matching relationship, the first modal data and the second modal data are synchronized.

[0005] Secondly, embodiments of this application provide a cross-modal data synchronization device, comprising: The acquisition module is configured to acquire first modality data and second modality data; The extraction module is configured to extract a first modal event from the first modal data and extract a second modal event from the second modal data; The determination module is configured to determine the matching relationship between the first modal event and the second modal event; The synchronization module is configured to synchronize the first modal data and the second modal data based on the matching relationship.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores an executable program, and the processor executes the executable program to perform the following steps: Acquire first modal data and second modal data; Extract a first modal event from the first modal data, and extract a second modal event from the second modal data; Determine the matching relationship between the first modal event and the second modal event; Based on the matching relationship, the first modal data and the second modal data are synchronized. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a flowchart illustrating a data synchronization method provided in one embodiment of this application; Figure 2 This is a flowchart of a semantic temporal extraction method provided in one embodiment of this application; Figure 3 This is a schematic diagram of cross-modal dynamic synchronization in a multi-person conference scenario provided by one embodiment of this application; Figure 4 This is a schematic diagram illustrating the dynamic change of time offset provided in one embodiment of this application; Figure 5 This is a schematic diagram of a data synchronization device provided in one embodiment of this application. Detailed Implementation

[0009] To enable those skilled in the art to better understand the technical solutions of the embodiments of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In multimodal data processing, accurate data synchronization is a prerequisite for applications such as speech separation, automatic speech recognition, and intelligent meeting analysis.

[0011] Existing cross-modal data synchronization schemes primarily rely on physical timestamps or multimedia container encapsulation timestamps for alignment. These methods synchronize data by directly comparing the absolute timestamps carried during data encapsulation. However, during the acquisition, transmission, and real-time playback of multimodal data, random fluctuations in network transmission latency, minute drifts in the device's sampling crystal oscillator, and processing delays in the underlying encoding buffer cause continuous dynamic time offsets in the data streams between modalities. Alignment methods that rely on fixed hardware-layer timestamps cannot effectively adapt to such dynamically changing environments, resulting in poor synchronization accuracy and robustness. like Figure 1 As shown in the figure, this application provides a data synchronization method, including: Step 101: Obtain the first modal data and the second modal data.

[0012] Step 102: Extract the first modal event from the first modal data, and extract the second modal event from the second modal data.

[0013] Step 103: Determine the matching relationship between the first modal event and the second modal event.

[0014] Step 104: Synchronize the first modal data and the second modal data based on the matching relationship.

[0015] This application acquires first-modal data and second-modal data, and extracts first-modal events and second-modal events with clear physical meaning from them respectively. This elevates the synchronization benchmark from the easily disturbed low-level signal features or fixed physical timestamps relied upon in existing technologies to an event dimension with high-level semantic understanding. Based on this, by determining the matching relationship between the first-modal events and second-modal events, an objective cross-modal correspondence anchor point is constructed at the physical action level. Furthermore, the synchronization of the first-modal data and second-modal data is directly driven based on the matching relationship. This means the entire synchronization process is no longer limited to a single time benchmark, but can be dynamically calibrated and corrected using newly generated semantic events and their matching relationships as the data stream continues to advance. This effectively overcomes the technical shortcomings of traditional methods, such as poor robustness in complex dynamic environments and cumulative error drift over long periods. Simultaneously, since this method only relies on the discretized representation of semantic events and the calculation of matching relationships, it does not involve high-load deep learning sequence processing, offering advantages such as low computational requirements, fast response speed, and ease of real-time deployment on edge computing devices.

[0016] The following is a detailed description of each step in the above data synchronization method in a specific embodiment.

[0017] Receiving or reading at least two independent sets of modal data that need to be synchronized. The first modal data and the second modal data refer to data with different physical acquisition attributes or signal characteristics. For example: audio data and video data, inertial sensor data and visual image data, text data and voice data, etc.

[0018] The data acquisition method can be flexibly selected according to the actual application scenario and is not limited to specific hardware interfaces. For example, real-time data streams can be received through sensor acquisition devices such as microphone arrays and cameras; or pre-recorded or packaged multimedia file data can be read from local storage media. In practical applications, preprocessing operations such as noise reduction, deduplication, and frame rate adjustment can be performed on the acquired first and second modal data to improve the accuracy of subsequent event extraction.

[0019] An event is a discrete action state that has a significant physical change or a specific semantic meaning. For example, in audio data, an event may be manifested as someone starting to utter a syllable, or a sudden increase or decrease in the energy of the voice; in video data, an event may be manifested as the opening and closing of a speaker's lips, or the turning of the head.

[0020] Due to the significant differences in the physical characteristics of modal data, the specific algorithms for event extraction can be diverse and are not limited to a single algorithm. Specifically, for audio data, methods such as calculating short-time energy difference, energy difference detection, and high-frequency transient energy proportion can be used to detect the start, abrupt changes, or specific phoneme features of sound, thereby extracting multiple discrete moments in the audio data as syllable start events, plosive events, or speaker switching events. For video data, facial keypoint detection and optical flow methods can be used to calculate the lip region motion vector to determine the opening and closing state of the lips, the movement trajectory of the head, or changes in facial orientation, thereby extracting lip opening and closing events, head movement change events, or facial orientation change events. Alternatively, trained audio classification neural networks or video action recognition neural networks can be used to directly output predicted labels and probabilities of specific actions in the data stream, marking positions with probabilities exceeding a preset threshold as events. The extraction of first-modal events and second-modal events is independent of each other, maintaining the original data frame timestamps unchanged on the timeline.

[0021] like Figure 2 As shown, the audio modality side extracts semantic events from the first modality data (i.e., audio data) using an audio semantic event extraction module, obtaining first modality semantic events. The first modality semantic events include at least one of audio energy mutation events, syllable initiation events, plosive events, and speaker switching events. The video modality side extracts semantic events from the second modality data (i.e., video data) using a video semantic event extraction module, obtaining second modality semantic events. The second modality semantic events include at least one of lip opening / closing events, head movement change events, and facial orientation change events. The event extraction processes of the audio and video modal sides are independent of each other, each recording the timestamps corresponding to its extracted modality semantic events as the data basis for subsequently determining the matching relationships between different modality semantic events.

[0022] Since the first modal event and the second modal event record the behavior of the same physical world under different sensing dimensions, by establishing a matching relationship between the events, the two independent data streams can be bound together in terms of physical action and physical time.

[0023] As a specific implementation method, a dual screening strategy combining temporal neighborhood constraints and feature similarity verification can be adopted. Specifically, firstly, a preset time threshold window is set with the timestamp of the first modal event as the center, and candidate events that occur within this window are screened from the second modal events, thereby eliminating interference items that are completely unrelated in physical time. Then, feature vectors of the first modal event and its candidate events are extracted respectively, such as the Mel frequency cepstral coefficient features of audio data and the optical flow motion vector of the lip region in video data, and the feature similarity between the two is calculated. When the calculated feature similarity is greater than the preset similarity threshold, and the event type of the candidate event satisfies the preset semantic mapping relationship, for example, the syllable start event in audio corresponds to the lip opening event in video, the system confirms that the first modal event and the candidate event constitute a matching relationship.

[0024] Understandably, the methods for determining matching relationships are not limited to the aforementioned combination of time thresholds and feature similarity. In practical applications, a deep learning-based cross-modal attention mechanism can also be used. The feature sequences of two sets of events are input into a cross-modal attention network, which adaptively allocates attention weights and directly outputs the alignment probability matrix between the event sequences. Alternatively, ignoring the individual differences of isolated events, the two modal data segments are treated as a whole sequence for cross-correlation calculation, and the matching relationship of the events is determined in reverse at the peak position of the cross-correlation function.

[0025] Specifically, using the matching relationship determined in step 103, the time axes of the first modal data and the second modal data are aligned. Since the matching relationship contains accurate timestamp pairs corresponding to the same physical action in different modalities, this step can determine the timing deviation between the two data streams and use this deviation information to guide the playback or processing progress of the data streams.

[0026] In this embodiment, a timeline forced mapping strategy can be used to perform synchronization. Specifically, the timestamps of each matching event in the first modality data are used as the target reference, and the timestamps of the corresponding matching events in the second modality data are mapped onto the reference. By dynamically inserting or dropping frames between adjacent matching points, the two data streams can be precisely aligned at the matching points, thereby completing the overall time alignment without having to explicitly calculate the offset.

[0027] As another specific implementation, the time offset between the first modal data and the second modal data can be calculated based on the matching relationship. This time offset is then converted into actual control commands. By dynamically controlling the dwell time of a certain modal data in the playback buffer, or increasing / decreasing the output delay, the output times of the two data streams can be made to converge. Those skilled in the art will understand that the specific physical means of synchronization described above can be flexibly adjusted according to the application scenario. In real-time streaming media transmission scenarios, waveform similarity overlap and addition methods can be used to stretch or compress the playback duration of audio frames; in offline editing scenarios, the encapsulation timestamp of the multimedia file can be directly modified to generate a synchronized new file; in non-multimedia data synchronization scenarios, precise hardware-level synchronization can also be achieved by adjusting the start time offset of the underlying sensor sampling window.

[0028] This application, through the acquisition and preprocessing of multimodal data, enables the method to be compatible with heterogeneous modal data with different physical acquisition attributes, such as audio, video, inertial sensor data, and text, thereby improving the universality and application breadth of the technical solution. Preprocessing effectively improves the accuracy of subsequent event extraction. Through semantic event extraction, the originally continuous and redundant raw signals are precisely extracted into discrete semantic events with significant physical abrupt changes or specific semantic meanings (such as syllable initiation and plosives in audio, and lip opening / closing and head movements in video). Furthermore, the extraction processes for the audio modality and the video modality are independent of each other and retain their respective original timestamps, ensuring that the data features between modalities do not interfere with each other and that the time reference is authentic, providing a precise data foundation for subsequent cross-modal matching.

[0029] By employing a dual screening strategy that combines temporal neighborhood constraints with feature similarity verification, the system effectively eliminates interference items that are completely irrelevant in physical time using a time threshold window. Furthermore, it overcomes the drawback of false positive matching caused by relying solely on temporal proximity by using features such as Mel frequency cepstral coefficients or optical flow motion vectors for feature similarity calculation, as well as joint judgment based on preset semantic mapping relationships. This significantly improves the accuracy and anti-interference ability of cross-modal event matching. At the same time, by using alternative implementation methods such as enumerating cross-correlation calculations or cross-modal attention mechanisms, the matching strategy is endowed with extremely strong algorithmic flexibility.

[0030] Based on the established matching relationship, fast and accurate synchronization can be achieved without explicitly calculating the time offset through forced timeline mapping with timestamp as the target reference and dynamic frame insertion or dropping operations. Alternatively, the actual time offset can be calculated and dynamically adjusted by combining playback buffer output delay control or waveform similarity overlap addition method. Different physical execution methods are applicable in various application scenarios such as real-time streaming media and offline editing, thereby comprehensively ensuring the temporal consistency of cross-modal data during playback, rendering or sensor sampling, significantly improving the adaptability and practical engineering feasibility of this synchronization solution.

[0031] In one embodiment of this application, determining the matching relationship between a first modal event and a second modal event includes: Extract the first event features from the first modal event, and extract the second event features from the second modal event; Based on the first event features and the second event features, the matching relationship between the first modal event and the second modal event is determined.

[0032] Because the physical properties of the first and second modes differ, the specific content of event features also varies from mode to mode. For audio data, the first event feature typically refers to the acoustic feature vector obtained after signal processing of a small segment of audio signal taken before or after the event timestamp, such as Mel frequency cepstral coefficients, linear prediction coefficients, or Mel spectrogram energy values. These features can accurately describe the spectral distribution and energy profile of the sound at that moment. For video data, the second event feature refers to the lip movement vector, facial region pixel change rate, or head posture angle data calculated by computer vision algorithms in the video frame corresponding to the event timestamp, used to describe the physical motion state of the scene at that moment. The extraction of event features is independent of the event label; it depends on the original data slice at the moment the event occurs. Therefore, it can objectively reflect the microscopic details of the physical behavior, providing the underlying data foundation for fine-grained cross-modal comparisons.

[0033] In this embodiment, event features essentially refer to key semantic anchors located within each modal event, and their core function is to achieve precise alignment of cross-modal data on the physical timeline. Specifically, the first event feature in the audio data corresponds to specific physical acoustic actions in the audio semantic event, such as the start and end of vocalization, the beginning of a syllable, or the moment of a plosive. These features can accurately mark the key positions of the audio signal on the timeline. The second event feature in the video data corresponds to specific visual motion actions in the video semantic event, such as lips opening, lips closing, the start of head rotation, or changes in facial orientation. These features can accurately mark the key positions of the video frame on the timeline.

[0034] The system calculates the similarity between two sets of extracted feature vectors to determine whether they physically point to the same action. Specifically, it calculates the similarity between the "vocal initiation" feature vector extracted from the audio and the "lip opening" feature vector extracted from the video; similarly, it calculates the similarity between the "vocal initiation" feature vector in the audio and the "lip closing" feature vector in the video. If the similarity of the feature vectors exceeds a preset similarity threshold, the two events are considered a match; otherwise, they are considered a mismatch. Commonly used metrics include cosine similarity, Euclidean distance, or Pearson correlation coefficient. By precisely mapping the vocal initiation and initiation features in the audio data to the lip opening and closing features in the video data, the system can bind these two key semantic anchors at the physical action level, thus providing an accurate and quantifiable reference benchmark for subsequent synchronization of the timelines of the first and second modal data based on matching relationships.

[0035] This application can effectively overcome the false matching problem that existing technologies can easily cause when relying solely on temporal proximity in complex scenarios, such as when the action times of different speakers in a multi-person conference happen to be close or overlap. It significantly improves the accuracy and robustness of cross-modal event matching operations.

[0036] In one embodiment of this application, determining the matching relationship between a first modal event and a second modal event based on a first event feature and a second event feature includes: Based on the first temporal features of each first modal event, candidate events corresponding to the first modal events are determined. The candidate events are those whose second temporal features match the first temporal features among multiple second modal events. The temporal features characterize the occurrence time of the modal events. Based on the matching degree between the first event features of the first modal event and the second event features of the candidate events, the second modal event that matches the first modal event is determined from the candidate events.

[0037] Specifically, a temporal neighborhood threshold can be preset, and the absolute time difference between the timestamp of the first modal event and the timestamp of each second modal event can be calculated. If the difference is less than the temporal neighborhood threshold, the second modal event is determined to be close to the current first modal event in physical time, and it is marked as a candidate event and added to the candidate list; if the difference is greater than or equal to the threshold, the second modal event is directly discarded and not included in the subsequent feature comparison process. Therefore, the candidate events are essentially a set of potential matching objects after initial screening based on the time condition.

[0038] In alternative implementations, the time neighborhood threshold can be dynamically and adaptively adjusted based on the real-time network jitter or historical matching success probability of the first and second modal data streams; alternatively, a multi-time window strategy can be adopted, which simultaneously presets multiple time windows of different scales, searches for candidate events within each time window, and takes the intersection as the final candidate event, thereby ensuring both recall and anti-interference capability.

[0039] For each candidate event, the feature similarity between the first event feature of the first modality event and the second event feature of the candidate event is calculated. This similarity value is the matching degree of this application. A matching degree threshold is preset. The matching degrees of all candidate events are sorted, and the candidate event with the highest matching degree that exceeds the matching degree threshold is selected as the second modality event that matches the first modality event. If the matching degree of all candidate events does not reach the preset threshold, it is determined that the current first modality event has not found a corresponding matching event, and no matching relationship is established.

[0040] The number of second-modal events can be quite large. Calculating feature similarity between each first-modal event and all second-modal events would incur a huge computational burden. The core of this step lies in using only the occurrence time (i.e., timestamp) of the modal events as a preliminary criterion, without calculating any numerical feature vectors, to quickly select a very small number of physically adjacent candidate events from a massive number of second-modal events, thereby significantly reducing the data scale for subsequent feature comparisons.

[0041] In one embodiment of this application, determining a second modal event matching the first modal event from the candidate events based on the matching degree between a first event feature of the first modal event and a second event feature of the candidate events includes: The matching degree between the first event feature and each second event feature in the candidate events is determined sequentially; If the matching degree between the semantic behavior represented by the first event feature and the semantic behavior represented by the second event feature satisfies the target matching condition, it is determined that the second modal event corresponding to the second event feature matches the first modal event.

[0042] Specifically, the second event features of each candidate event can be obtained one by one in the predetermined order of the candidate queue, and then compared and calculated with the current first event features.

[0043] After calculating the matching degree sequentially, the system first filters out pairings that have high feature similarity but do not meet the preset semantic mapping relationship. For example, in a multi-person conference scenario, the system may mistakenly calculate the audio feature of "someone speaking" as highly similar to the video feature of "another person turning their head" due to noise interference. In this case, the pairing is discarded because it does not meet the semantic mapping condition.

[0044] Semantic mapping clarifies the correspondence between audio events and video events. For example, a syllable start event corresponds to a lip opening event, and a syllable end event corresponds to a lip closing event.

[0045] Only when the matching degree calculation result of a candidate event exceeds the preset similarity threshold, and the event type of the candidate event satisfies the above semantic mapping relationship with the first modal event, is it determined to meet the target matching condition, and finally it is determined that the second modal event corresponding to the candidate event constitutes a matching relationship with the current first modal event.

[0046] In cross-modal matching, this application's embodiments no longer rely solely on the similarity of feature vectors, but instead combine feature similarity with the semantic mapping relationship of physical actions as a dual criterion for judgment. This mechanism can effectively eliminate false positive matches caused by accidental similarity of low-level features, thereby significantly improving the accuracy and anti-interference ability of cross-modal event matching in complex interaction scenarios, ensuring that the final matching result truly conforms to objective physical facts.

[0047] In one embodiment of this application, both the first modal event and the second modal event include events generated by different users; Determining the matching relationship between the first modal event and the second modal event includes: Based on user characteristics, determine the first modal events and second modal events generated by each user; The first modal event generated by each user is determined sequentially, along with its matching second modal event.

[0048] In practical applications, multimedia data streams are not always generated by a single object. For example, in video conferencing, roundtable discussions, or multi-person interviews, the audio data stream may contain the voices of multiple people at the same time, and the video data stream may also record the lips and head movements of multiple people at the same time.

[0049] The first modal event may consist of a mixture of speaker A's speech, speaker B's speech, and background noise events; the second modal event may consist of a mixture of speaker A's lip opening and closing, speaker B's lip opening and closing, and head movements of other participants. Although these events may highly overlap in time, they belong to different user sources in terms of physical attributes.

[0050] The user features here are unique identifiers that distinguish different users. These could be unique voiceprint features, facial features, or the user's spatial location. Using these features, the first modality events are clustered by user affiliation into the audio event set of user A, the audio event set of user B, and so on. Similarly, the second modality events are clustered by user affiliation into the video event set of user A, the video event set of user B, and so on.

[0051] For audio events, their corresponding acoustic user features can be extracted; for video events, their corresponding visual user features can be extracted. By calculating the clustering distance between feature vectors, the first modality event with a high concentration of feature vector similarity is labeled as an event generated by user A, and the second modality event with a high concentration of feature vector similarity is also labeled as an event generated by user A.

[0052] This application embodiment utilizes user characteristics such as voiceprint, vision, or spatial orientation to decouple and separate mixed event streams according to identity attributes, thereby effectively eliminating event crossover and confusion caused by the high overlap of different user actions in physical time in multi-user scenarios.

[0053] like Figure 3 As shown, in this scenario, both the acquired audio and video data contain events generated by users A and B. Based on user characteristics, the original audio and video data streams are parsed into audio and video data for speaker A and speaker B, respectively, and then input into the semantic event synchronization engine. The semantic event synchronization engine processes the audio and video data of each user in parallel or in time-sharing, and outputs the synchronization results for speaker A and speaker B, respectively, based on the processing results.

[0054] In one embodiment of this application, synchronizing first modal data and second modal data based on a matching relationship includes: Based on the matching relationship, the time offset between the first modal data and the corresponding second modal data is determined; Based on the time offset, the first modal data and the second modal data are synchronized.

[0055] Extract the timestamp T of the first modal event corresponding to the matching relationship from the matching relationship. A Second modal event timestamp T B Then, calculate the time difference ΔT = T B T A Since multiple matching relationships may be generated in actual operation, a smoothing algorithm such as weighted averaging, median filtering, or Kalman filtering can be used to calculate the time difference of all matching relationships to eliminate the instantaneous jitter caused by single matching errors, and finally obtain a stable and reliable time offset between the first modality data and the second modality data within the current time window.

[0056] When the time offset indicates that the first modal data is ahead, the playback duration of the first modal data is slowed down by increasing the residence time of the first modal data in the buffer queue or by using a waveform similarity overlap and addition algorithm to stretch the playback duration of the first modal data, so that the first modal data catches up with the second modal data.

[0057] When the offset indicates that the second modality data is ahead, the rendering progress of the second modality data is adjusted by increasing the rendering delay of the second modality data, repeating the rendering of the previous frame, or discarding some consecutive frames, so that the two eventually maintain temporal consistency at the matching point.

[0058] This application's embodiments transform the established cross-matching relationship into a computer-quantifiable time offset, enabling the system to intuitively and accurately obtain the relative deviation of two independent data streams on the overall time axis, providing a clear numerical basis for subsequent physical correction. Based on this, the quantified deviation index is used as a direct decision variable to drive playback control, cache scheduling, or data presentation adjustments, allowing the system to precisely guide the output or rendering progress of modal data such as audio and video based on the magnitude and direction of the measured time difference.

[0059] In one embodiment of this application, synchronizing first modal data and second modal data based on a time offset includes: In response to the time offset exceeding the offset threshold, perform at least one of the following steps based on the time offset: Control the output delay of the first modal data in the playback buffer to compensate for the time offset; Control the output delay of the second modal data in the playback buffer to compensate for the time offset.

[0060] When the time offset exceeds the offset threshold, the output timing of the first modality data can be controlled by directly manipulating the playback buffer. The playback buffer stores decoded data that has not yet been sent to the speaker or screen. Controlling its output delay essentially changes the time the data stays in the buffer, thereby delaying or advancing the output. When the time offset exceeds the offset threshold, the output rhythm of the second modality data can also be adjusted by manipulating the playback buffer to compensate for the time offset.

[0061] This application introduces a trigger decision mechanism based on an offset threshold, which ensures that physical correction is performed only when the deviation exceeds the tolerance, avoiding frequent adjustments and playback abnormalities caused by short-term minor jitter, thereby effectively improving the stability of synchronous actions and user experience.

[0062] In one embodiment of this application, the method further includes: Based on the time offset, offset compensation is performed on the time features in the matching relationship; Calculate the time offset for the next period based on the timestamp in the matching relationship after offset compensation and the timestamp in the matching relationship determined for the next period. Based on the time offset of the next cycle, perform synchronization for the next cycle.

[0063] However, after the previous round of calculations, the timestamps carried by the matching relationships no longer reflect the latest physical time reference. Without compensation, these old anchor points will experience reference drift in the next round of calculations. Therefore, offset compensation is performed based on the currently executed time offset to realign the historical anchor points with the real-time physical timeline.

[0064] For the timestamps of the first and second modal events in the matching relationship, the following operation is performed: the timestamp value is directly added to (or subtracted from) by the time offset. For example, to compensate for the offset, the first modal data is delayed by 50ms. This will uniformly increase the timestamps of all first modal events in the matching relationship by 50ms, for example, updating 12.30 seconds to 12.35 seconds, thus ensuring that the physical occurrence time pointed to by this matching relationship perfectly matches the latest system playback benchmark.

[0065] After entering a new calculation cycle, the old matching relationships that have completed baseline compensation are fused with the new matching relationships confirmed by the current cycle's matching algorithm. For example, the timestamp differences corresponding to the two sets of matching relationships are fed into a weighted moving average filter or a Kalman filter. This filter assigns higher weight to the current new data and, combined with the data from the old matching relationships, calculates a comprehensive time offset that reflects both current sudden fluctuations and historical dynamic trends.

[0066] like Figure 4 As shown, the horizontal axis represents the running time period (T1 to T5), and the vertical axis represents the time offset (ms). This curve visually illustrates the dynamic trend of the time offset between the first modal data and the second modal data calculated in real time within different running periods. When the time offset of a certain period exceeds a preset offset threshold, in response to the offset exceeding the threshold, the playback buffer output delay is adjusted based on the current time offset to compensate for the time offset. Simultaneously, offset compensation is applied to the time features in the confirmed matching relationships based on this time offset, so that the time base of the offset-compensated historical matching relationships is realigned with the current physical time axis. In the next time period, the smoothed time offset for the next period is calculated based on the timestamps in the offset-compensated historical matching relationships and the timestamps in the newly determined matching relationships, and synchronization for the next period is performed accordingly.

[0067] This application's embodiments no longer rely solely on single burst time offsets for correction. Instead, they dynamically adjust the playback buffer output delay of the first and / or second modal data for the next cycle based on long-term smooth tracking trend values. In this way, accumulated errors that may occur during long-term operation can be eliminated at their source, maintaining long-term alignment of audio, video, or sensor data.

[0068] Through the aforementioned closed-loop correction strategy based on long-term smoothing trends, this application can maintain high-precision synchronization between the first modality data and the second modality data over a long period. On the one hand, this alignment can fundamentally eliminate the lip-sync issues caused by misalignment of the audio-visual timeline, significantly improving the user's subjective audiovisual experience and information intelligibility in actual playback or real-time interactive scenarios. On the other hand, maintaining consistent multimodal data over a long period can provide a highly coupled and stable input benchmark for downstream algorithms such as automatic speech recognition and target speech separation. Furthermore, since the correction process is driven by a smoothing trend rather than a single burst error, it can effectively avoid stuttering or screen tearing caused by frequent adjustments to buffer output latency.

[0069] like Figure 5 As shown, this application embodiment provides a cross-modal data synchronization device, including: The acquisition module 501 is configured to acquire first modal data and second modal data; Extraction module 502 is configured to extract a first modal event from the first modal data and extract a second modal event from the second modal data; The module 503 is configured to determine the matching relationship between the first modal event and the second modal event; Synchronization module 504 is configured to synchronize first modal data and second modal data based on matching relationships.

[0070] In one embodiment of this application, the determining module 503 is configured to extract a first event feature from a first modal event, extract a second event feature from a second modal event, and determine a matching relationship between the first modal event and the second modal event based on the first event feature and the second event feature.

[0071] In one embodiment of this application, the determining module 503 is configured to determine candidate events corresponding to the first modal events based on the first time features of each first modal event. The candidate events are events among a plurality of second modal events whose second time features match the first time features. The time features characterize the occurrence time of the modal event. Based on the matching degree between the first event features of the first modal event and the second event features of the candidate events, the second modal event matching the first modal event is determined from the candidate events.

[0072] In one embodiment of this application, the determining module 503 is configured to sequentially determine the matching degree between the first event feature and each second event feature in the candidate events; if the matching degree between the semantic behavior represented by the first event feature and the semantic behavior represented by the second event feature satisfies the target matching condition, the second modal event corresponding to the second event feature is determined to match the first modal event.

[0073] In one embodiment of this application, the first modal event and the second modal event both include events generated by different users; the determining module 503 is configured to determine the first modal event and the second modal event generated by each user based on user characteristics; and sequentially determine the first modal event generated by each user and its matching second modal event.

[0074] In one embodiment of this application, the synchronization module 504 is configured to determine the time offset between the first modal data and the corresponding second modal data based on the matching relationship; and to synchronize the first modal data and the second modal data based on the time offset.

[0075] In one embodiment of this application, the synchronization module 504 is configured to, in response to a time offset exceeding an offset threshold, perform at least one of the following steps based on the time offset: control the output delay of the first modal data in the playback buffer to compensate for the time offset; control the output delay of the second modal data in the playback buffer to compensate for the time offset.

[0076] In one embodiment of this application, the synchronization module 504 is configured to perform offset compensation on the time features in the matching relationship based on the time offset; calculate the time offset of the next period based on the timestamp in the offset-compensated matching relationship and the timestamp in the matching relationship determined in the next period; and perform synchronization of the next period based on the time offset of the next period.

[0077] This application provides an electronic device, including a memory and a processor. The memory stores an executable program, and the processor executes the executable program to perform the following steps: Acquire first modal data and second modal data; Extract the first modal event from the first modal data, and extract the second modal event from the second modal data; Determine the matching relationship between the first modal event and the second modal event; Based on the matching relationship, the first modality data and the second modality data are synchronized.

[0078] This application provides a computer program product that, when executed by a processor, implements the methods of any of the above embodiments.

[0079] It should be understood that in the embodiments of this application, the processor may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0080] It should also be understood that the memory mentioned in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Specifically, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0081] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0082] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0083] In addition to the data bus, this bus may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled "bus" in the diagram.

[0084] It should also be understood that the first, second, third, fourth and various numerical designations used herein are merely for descriptive convenience and are not intended to limit the scope of this application.

[0085] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0086] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0087] In the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0088] Those skilled in the art will recognize that the various illustrative logical blocks (ILBs) and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0092] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data synchronization method, comprising: Acquire first modal data and second modal data; Extract a first modal event from the first modal data, and extract a second modal event from the second modal data; Determine the matching relationship between the first modal event and the second modal event; Based on the matching relationship, the first modal data and the second modal data are synchronized.

2. The method as described in claim 1, Determining the matching relationship between the first modal event and the second modal event includes: Extract a first event feature from the first modal event, and extract a second event feature from the second modal event; Based on the first event features and the second event features, the matching relationship between the first modal event and the second modal event is determined.

3. The method as described in claim 2, Based on the first event features and the second event features, the matching relationship between the first modal event and the second modal event is determined, including: Based on the first time features of each first modal event, candidate events corresponding to the first modal events are determined. The candidate events are events among a plurality of second modal events whose second time features match the first time features. The time features characterize the occurrence time of the modal event. Based on the matching degree between the first event feature of the first modal event and the second event feature of the candidate events, a second modal event matching the first modal event is determined from the candidate events.

4. The method as described in claim 3, Based on the matching degree between the first event feature of the first modal event and the second event feature of the candidate events, a second modal event matching the first modal event is determined from the candidate events, including: The matching degree between the first event feature and each second event feature in the candidate events is determined sequentially; If the matching degree between the semantic behavior represented by the first event feature and the semantic behavior represented by the second event feature satisfies the target matching condition, it is determined that the second modal event corresponding to the second event feature matches the first modal event.

5. The method as described in claim 1, in, Both the first modal event and the second modal event contain events generated by different users; Determining the matching relationship between the first modal event and the second modal event includes: Based on user characteristics, determine the first modal events and second modal events generated by each user; The first modal event generated by each user is sequentially determined and its matching second modal event is determined.

6. The method as described in any one of claims 1-5, Based on the matching relationship, synchronizing the first modal data and the second modal data includes: Based on the matching relationship, the time offset between the first modal data and the corresponding second modal data is determined; Based on the time offset, the first modal data and the second modal data are synchronized.

7. The method as described in claim 6, Based on the time offset, synchronizing the first modal data and the second modal data includes: In response to the time offset exceeding an offset threshold, at least one of the following steps is performed based on the time offset: Control the output delay of the first modal data in the playback buffer to compensate for the time offset; The output delay of the second modal data in the playback buffer is controlled to compensate for the time offset.

8. The method of claim 6, further comprising: Based on the time offset, offset compensation is performed on the time features in the matching relationship; The time offset of the next period is calculated based on the timestamp in the matching relationship after offset compensation and the timestamp in the matching relationship determined in the next period. Based on the time offset of the next cycle, the synchronization of the next cycle is performed.

9. A cross-modal data synchronization device, comprising: The acquisition module is configured to acquire first modality data and second modality data; The extraction module is configured to extract a first modal event from the first modal data and extract a second modal event from the second modal data; The determination module is configured to determine the matching relationship between the first modal event and the second modal event; The synchronization module is configured to synchronize the first modal data and the second modal data based on the matching relationship.

10. An electronic device comprising a memory and a processor, the memory storing an executable program, the processor executing the executable program to perform the following steps: Acquire first modal data and second modal data; Extract a first modal event from the first modal data, and extract a second modal event from the second modal data; Determine the matching relationship between the first modal event and the second modal event; Based on the matching relationship, the first modal data and the second modal data are synchronized.

Images