Method for synchronizing multimedia data and multimedia synchronization system

Abstract

This application discloses a multimedia data synchronization method and a multimedia synchronization system. A first device transmits first multimedia data to a second device through a first transmission channel at a first moment. By introducing a secondary transmission mechanism with a preset duration based on the time taken for the second device to complete AI processing, the second multimedia data, originally transmitted synchronously with the first multimedia data, is converted into a delayed transmission data stream. This enables the multimedia synchronization system to support latency adaptation within a second-level or even longer range, matching the latency characteristics of the AI ​​processing link. Based on this, after receiving the second multimedia data, the second device aligns the timing-matched second multimedia data with the processed multimedia data and outputs them synchronously, ensuring that the second multimedia data and the processed multimedia data are consistent in time. Therefore, it effectively solves the problem of audio and video asynchrony in AI multimedia processing application scenarios.

Description

Technical Field

[0001] This application relates to multimedia technology, including but not limited to a method and system for synchronizing multimedia data. Background Technology

[0002] In multimedia data processing, taking audio and video signals as examples, different types of delays occur in their processing and transmission paths, causing a discrepancy in their arrival time at the user's senses, i.e., audio and video desynchronization. With the deepening application of artificial intelligence (AI) technology in the multimedia field, in addition to different types of delays, delays of 3 to 5 seconds or even longer may be introduced due to contextual association and semantic understanding, making audio and video desynchronization even more severe.

[0003] In related technologies, buffering audio and / or video signals is commonly used to adjust latency, typically within a few hundred milliseconds. For example, the Audio Return Channel (ARC) or Enhanced Audio Return Channel (EARC) protocol of High-Definition Multimedia Interface (HDMI) usually supports latency synchronization adjustment within the range of 0 to 255 milliseconds.

[0004] However, since the adjustable range of related technologies is only a few hundred milliseconds, it cannot handle latency at the second level. Therefore, the problem of audio and video desynchronization is particularly prominent in AI multimedia processing applications. Summary of the Invention

[0005] In view of this, the multimedia data synchronization method and multimedia synchronization system provided in this application can support second-level delay adjustment, solving the problem of audio and video desynchronization in AI multimedia processing application scenarios. The multimedia data synchronization method and multimedia synchronization system provided in this application are implemented as follows: This application provides a multimedia data synchronization method, which is applied to a multimedia synchronization system. The multimedia synchronization system includes a first device and a second device. The second device includes an AI processing unit for performing AI processing on multimedia data. The AI ​​processing includes translating audio data into text subtitles or recognizing image data to generate text tags. The first device and the second device are connected via a first transmission channel and a second transmission channel. The synchronization method includes: The first device transmits first multimedia data to the second device through the first transmission channel at a first moment; The second device performs AI processing on the first multimedia data through the AI ​​processing unit to obtain processed multimedia data; The first device transmits second multimedia data to the second device through the second transmission channel at a second time. The data content of the second multimedia data is the same as the data content of the first multimedia data. The second time is later than the first time and is separated from the first time by a preset time. The preset time is related to the time taken for the second device to complete the AI ​​processing. The second device aligns the second multimedia data and the processed multimedia data, and outputs the second multimedia data and the processed multimedia data synchronously.

[0006] In this embodiment, the first device transmits first multimedia data to the second device through a first transmission channel at a first moment; the second device performs AI processing on the first multimedia data through an AI processing device to obtain processed multimedia data; the first device transmits second multimedia data to the second device through a second transmission channel at a second moment, the data content of the second multimedia data being the same as the data content of the first multimedia data, the second moment being later than the first moment, and the time interval between the second moment and the first moment being a preset duration, the preset duration being related to the time taken for the second device to complete the AI ​​processing; the second device aligns the second multimedia data and the processed multimedia data, and synchronously outputs the second multimedia data and the processed multimedia data.

[0007] Based on this, the first device transmits the first multimedia data to the second device through the first transmission channel at the first moment, enabling the second device to receive the first multimedia data in advance and perform AI processing. By introducing a secondary transmission mechanism with a preset duration based on the time taken for the second device to complete AI processing, the second multimedia data, which was originally transmitted synchronously with the first multimedia data, is converted into a data stream with a preset delay. This allows the multimedia synchronization system to support latency adaptation within a second or even longer range, matching the latency characteristics of the AI ​​processing link. Furthermore, after receiving the second multimedia data, the second device aligns the timing-matched second multimedia data with the processed multimedia data and outputs them synchronously, ensuring that the second multimedia data and the processed multimedia data are consistent in time. Thus, the problem of audio and video asynchrony in AI multimedia processing application scenarios can be effectively solved.

[0008] In some possible embodiments, the second device aligns the second multimedia data and the processed multimedia data, and synchronously outputs the second multimedia data and the processed multimedia data, including: The second device acquires the first time identifier information of the first multimedia data and the second time identifier information of the second multimedia data respectively; the first time identifier information is used to characterize the first acquisition time of the first multimedia data, and the second time identifier information is used to characterize the second acquisition time of the second multimedia data. Based on the first time identifier information and the second time identifier information, the second multimedia data and the processed multimedia data are aligned, and the second multimedia data and the processed multimedia data are output synchronously.

[0009] In this embodiment, the second device acquires first time identifier information of the first multimedia data and second time identifier information corresponding to the second multimedia data. Based on this, the second device aligns the second multimedia data with the processed multimedia data using the first and second time identifier information as time references, ensuring that the second multimedia data and the processed multimedia data are time-consistent. This achieves synchronized output of the second multimedia data and the processed multimedia data. By using time identifier information for alignment calibration, timing offsets caused by transmission delay fluctuations, processing time differences, and other factors can be avoided, improving the accuracy and stability of audio and video synchronization in AI multimedia processing scenarios.

[0010] In some possible embodiments, aligning the second multimedia data and the processed multimedia data according to the first time identifier information and the second time identifier information, and synchronously outputting the second multimedia data and the processed multimedia data, includes: If the first acquisition time represented by the first time identifier information is greater than the second acquisition time represented by the second time identifier information, after waiting for a first duration, the second multimedia data and the processed multimedia data are output. The first duration is the interval between the first acquisition time and the second acquisition time. When the first acquisition time represented by the first time identifier information is equal to the second acquisition time represented by the second time identifier information, the second multimedia data and the processed multimedia data are output in real time.

[0011] In this embodiment, when the first acquisition time represented by the first time identifier information is greater than the second acquisition time represented by the second time identifier information, the second device waits for a first duration before outputting the second multimedia data and the processed multimedia data; when the first acquisition time represented by the first time identifier information is equal to the second acquisition time represented by the second time identifier information, the second multimedia data and the processed multimedia data are output in real time. Therefore, based on the comparison result of the first and second time identifier information, differentiated output control of the second multimedia data and the processed multimedia data is achieved, ensuring that the second multimedia data and the processed multimedia data remain consistent in time, enabling the processed multimedia data to be aligned with the second multimedia data.

[0012] In some possible embodiments, the synchronization method further includes: When the first acquisition time represented by the first time identifier information is less than the second acquisition time represented by the second time identifier information, the second device performs a target operation on the target multimedia data. The target operation includes discarding the target multimedia data, which is the processed multimedia data corresponding to the first time identifier information.

[0013] In some possible embodiments, the second device aligns the second multimedia data and the processed multimedia data, and synchronously outputs the second multimedia data and the processed multimedia data, including: The second device obtains the processing time of the AI ​​processing device in performing AI processing on the first multimedia data; Based on the processing time and the preset time, the second multimedia data and the processed multimedia data are aligned and output synchronously.

[0014] In this embodiment, the second device acquires the processing time of the AI ​​processing device on the first multimedia data and combines it with a preset time to perform time alignment on the second multimedia data and the processed multimedia data, thereby achieving synchronous output of the second multimedia data and the processed multimedia data. By combining the processing time and the preset time for time alignment, the second device can correct the timing deviation caused by long AI processing in real time, ensuring that the second multimedia data and the processed multimedia data are precisely aligned on the timeline. This avoids over-buffering or data race-out caused by fixed delay strategies, significantly improving the accuracy and stability of audio and video synchronization in AI multimedia processing scenarios.

[0015] In some possible embodiments, aligning the second multimedia data and the processed multimedia data according to the processing time and the preset time, and synchronously outputting the second multimedia data and the processed multimedia data, includes: When the processing time is equal to the preset time, the second multimedia data and the processed multimedia data are output in real time. If the processing time is less than the preset time, the third moment when the AI ​​processing is completed is obtained, and after a second time from the third moment, the second multimedia data and the processed multimedia data are output, where the second time is the difference between the preset time and the processing time.

[0016] In this embodiment, when the processing time equals a preset time, the second multimedia data and the processed multimedia data are output in real time. When the processing time is less than the preset time, the third moment after the completion of AI processing is obtained, and from the third moment, after a second time period, the second multimedia data and the processed multimedia data are output. Therefore, based on the comparison between the processing time and the preset time, differentiated output control of the second multimedia data and the processed multimedia data is achieved, ensuring that the second multimedia data and the processed multimedia data remain consistent in time, allowing the processed multimedia data to be aligned with the second multimedia data.

[0017] In some possible embodiments, the synchronization method further includes: If the processing time exceeds the preset time, the second device discards the target multimedia data, which is the multimedia data corresponding to the processing time.

[0018] In some possible embodiments, the first device further includes a data acquisition device and a buffer device, wherein the first multimedia data is data acquired by the first device through the data acquisition device and stored in the buffer device, and after the second device performs a target operation on the target multimedia data, the synchronization method further includes: The second device transmits control commands to the first device, the control commands being used to instruct an increase in the cache space of the cache device; In response to the control command, the first device expands the capacity of the cache device.

[0019] In this embodiment, after the second device performs a target operation on the target multimedia data, it transmits a control command to the first device. In response to the control command, the first device expands the capacity of its buffer device, increasing its buffer space. Thus, the second device can achieve time alignment between the second multimedia data and the processed multimedia data during subsequent data processing.

[0020] In some possible embodiments, the first device further includes an image display device, and the second device further includes an audio playback device. After the second device aligns the second multimedia data and the processed multimedia data and synchronously outputs the second multimedia data and the processed multimedia data, the synchronization method further includes: The second device obtains target image data and target audio data based on the aligned second multimedia data and the processed multimedia data; The second device transmits the target image data to the first device; The first device displays the target image data through the image display device, and the second device plays the target audio data through the audio playback device.

[0021] In this embodiment, the second device obtains target image data and target audio data based on the aligned second multimedia data and the processed multimedia data. The second device transmits the target image data to the first device. The first device displays the target image data via an image display device, and the second device plays the target audio data via an audio playback device. Through the aforementioned echo mechanism, the first device displays the target image data via the image display device, and the second device plays the target audio data via the audio playback device, achieving precise synchronization of the target image data and target audio data on the time axis. This ensures reliable audio and video synchronization even in scenarios with significant transmission or processing delays, effectively improving the adaptability and flexibility of the multimedia synchronization system to meet the multimedia synchronization needs of complex AI application environments.

[0022] The multimedia synchronization system provided in this application includes a first device and a second device. The second device includes an AI processing device for AI processing of multimedia data. The AI ​​processing includes translating audio data into text subtitles or recognizing image data to generate text tags. The first device and the second device are connected through a first transmission channel and a second transmission channel. The first device is used to transmit first multimedia data to the second device through a first transmission channel at a first moment, wherein the data content of the first multimedia data is the same as the data content of the initial multimedia data. The second device is used to perform AI processing on the first multimedia data through the AI ​​processing device to obtain processed multimedia data; The first device is also configured to transmit second multimedia data to the second device through a second transmission channel at a second time. The data content of the second multimedia data is the same as the data content of the first multimedia data. The second time is later than the first time and is separated from the first time by a preset time interval. The preset time interval is related to the time taken for the second device to complete the AI ​​processing. The second device is further configured to align the second multimedia data and the processed multimedia data, and synchronously output the second multimedia data and the processed multimedia data.

[0023] The computer device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the synchronization method described in this application.

[0024] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the synchronization method provided in this application embodiment. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0026] Figure 1 This is a diagram illustrating an application scenario of the synchronization method provided in the embodiments of this application. Figure 2 This is a flowchart illustrating a synchronization method provided in an embodiment of this application; Figure 3 This is another flowchart illustrating the synchronization method provided in the embodiments of this application; Figure 4 A schematic diagram illustrating the relationship between the preset duration and the first and second moments provided in the embodiments of this application; Figure 5 Another flowchart illustrating the synchronization method provided in this application embodiment; Figure 6 This is a schematic diagram illustrating the increase of cache space provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0029] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0031] To better understand the synchronization method provided in the embodiments of this application, the synchronization methods in related technologies and the existing technical problems are first explained.

[0032] In existing multimedia systems, the main reason for audio and video asynchrony is that audio signals and video signals each experience different types of delays during processing and transmission.

[0033] Specifically, when terminal devices such as televisions and monitors process video signals, latency is introduced due to the decoding, deinterlacing, and image enhancement operations required. Compared to analog televisions, digital televisions exhibit significantly greater latency. For example, the latency in the video signal processing path of an analog television is approximately 1 / 60th of a second, while the latency of digital high-definition television (HDTV) is approximately 1 / 8 to 1 / 5th of a second.

[0034] Similarly, when terminal devices process audio signals, latency is introduced due to the decoding, digital signal processing (such as equalization and virtual surround), and digital-to-analog conversion operations. Typically, the processing latency of audio signals is less than that of video signals. For example, the latency of an audio signal processing path may be less than three frames, while the latency of a video signal processing path may reach six or more frames.

[0035] In addition, audio and video signals also introduce transmission latency when transmitted through channels such as wired HDMI or wireless Wi-Fi. This latency is further amplified, especially in high-resolution, high-refresh-rate transmission scenarios.

[0036] Due to video processing latency, audio processing latency, and transmission latency, there is a discrepancy in the timing of audio and video signals reaching the user's senses, resulting in audio-video desynchronization. In AI multimedia processing applications, in addition to latency in the processing and transmission paths, new latency is introduced by the characteristics and processing mechanisms of AI models, making audio-video desynchronization even more severe. Specifically: 1. Contextual Association Recognition Contextual understanding primarily relies on self-attention mechanisms, which have a time complexity of O(n²) (where n is the number of context tokens). This means that the computational cost of the AI ​​model increases non-linearly with the length of the context. For example, doubling the context length increases the computational cost of the AI ​​model by approximately four times. In scenarios involving long texts or multi-turn dialogues, when the context accumulates to thousands of tokens, attention matrix operations become the main source of time consumption, significantly increasing computational latency. In streaming or real-time translation scenarios, because decoding is required on the fly, the continuous calculation of the association between historical context and the current segment further increases processing latency.

[0037] 2. High data processing complexity AI models are typically deep learning models, which have a large number of parameters and high computational requirements. These models introduce computational latency due to numerous matrix operations and nonlinear transformations. For example, in video conferencing and media streaming scenarios, audio and video experience asymmetric latency in packet processing.

[0038] Data preprocessing stage: This stage typically requires preprocessing the data input to the AI ​​model, such as normalization, feature extraction, and data augmentation. Preprocessing also adds processing latency, especially noticeable when processing high-resolution images or videos. Furthermore, during audio and video data transmission, transmission delays may occur due to hardware limitations and the condition of the wireless air interface link.

[0039] Post-processing stage: This stage typically requires post-processing of the AI ​​model's output data, such as noise reduction, smoothing, and result parsing. Post-processing also adds additional processing latency. In HDMI-based audio / video systems, the difference in latency between the audio and video signal processing paths causes the displayed image and output audio to be out of sync.

[0040] 3. Insufficient algorithm optimization In terms of model optimization, although techniques such as model compression, pruning, and quantization can reduce computation and memory usage, they may affect the accuracy and real-time performance of the model to some extent.

[0041] In terms of parallel computing, accelerators such as GPUs and TPUs can significantly improve computing speed, but the algorithm needs to be modified and optimized for parallelization. If the optimization is insufficient, it may increase latency.

[0042] Regarding real-time optimization, some AI algorithms prioritize accuracy over real-time performance in their design, requiring special optimization to meet real-time requirements. For example, existing methods for resolving audio-video desynchronization include manually setting the audio delay duration at the HDMI transmitter or receiver. Another example is setting the processing time for image and audio at the receiver in the extended display identifier data of the playback device, according to the HDMI 1.3 standard. Yet another example is processing according to the HDMI 1.3 / 1.4 standard, where different resolutions of video and audio have different processing latencies.

[0043] 4. Hardware resource limitations In terms of computing power, AI applications have high requirements for computing power. If the hardware computing power is insufficient, it will lead to a decrease in computing speed and an increase in latency.

[0044] In terms of memory bandwidth, AI models typically require a significant amount of memory bandwidth for data reading and writing. Insufficient memory bandwidth can become a bottleneck for the system.

[0045] In terms of storage speed, the read and write speed of data from storage devices such as hard drives or SSDs will also increase latency.

[0046] 5. Security and Privacy Protection In terms of data encryption, encrypting and decrypting data increases the computational burden and leads to increased latency.

[0047] In terms of access control, secure access control mechanisms may also add additional latency.

[0048] In terms of privacy protection technologies, such as differential privacy and federated learning, while protecting privacy, may also increase computational complexity and processing latency.

[0049] In summary, AI applications not only experience delays in the processing and transmission paths, but also suffer from processing delays of several seconds or even longer due to a combination of factors, including contextual association recognition, high data processing complexity, insufficient algorithm optimization, hardware resource limitations, and security and privacy protection.

[0050] Because the adjustable range of the relevant technology is only a few hundred milliseconds, it cannot cope with second-level latency. As a result, there is a serious audio-video desynchronization problem in AI multimedia processing applications, which seriously affects the user experience.

[0051] In view of this, embodiments of this application provide a method for synchronizing multimedia data, which is applied to a multimedia synchronization system 100. Figure 1 This diagram illustrates an application scenario of the synchronization method provided in the embodiments of this application. For example... Figure 1 As shown, the multimedia synchronization system 100 includes a first device 10 and a second device 20. The second device 20 includes an AI processing unit for performing AI processing on multimedia data. For example, AI processing includes translating audio data into text subtitles or recognizing image data to generate text tags. The first device 10 and the second device 20 are connected via a first transmission channel and a second transmission channel. The first device 10 may include, but is not limited to, mobile phones, video conferencing terminals, live streaming devices, tablets, laptops, in-vehicle terminals, etc. The second device 20 may include, but is not limited to, cloud-based AI inference platforms, smart TVs, multimedia gateway devices, in-vehicle infotainment systems with AI processing, and smart speakers with AI processing. The functions implemented by this synchronization method can be achieved by the processors in the first device 10 and the second device 20 calling program code.

[0052] It should be noted that the first device 10 may or may not include a data acquisition device, and this application embodiment does not specifically limit this.

[0053] The following will further introduce the multimedia data synchronization method provided in the embodiments of this application, how to support second-level delay adjustment, and solve the problem of audio and video asynchrony in AI multimedia processing application scenarios.

[0054] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the synchronization method provided in an embodiment of this application. Figure 2 As shown, the synchronization method may include: S201, The first device transmits first multimedia data to the second device through the first transmission channel at the first moment.

[0055] The first transmission channel is used to transmit multimedia data for AI processing to the second device, prioritizing the scheduling of multimedia data that needs AI processing so that the second device can perform AI processing on the multimedia data in advance.

[0056] For example, multimedia data may include, but is not limited to, audio data, video data, and image data.

[0057] S202. The second device performs AI processing on the first multimedia data through an AI processing unit to obtain processed multimedia data.

[0058] AI processing includes translating audio data into text subtitles or recognizing image data to generate text tags.

[0059] For example, when the first multimedia data includes audio data, the AI ​​processing device converts the speech content in the audio data into text using a speech recognition model, and generates text subtitles corresponding to the speech content using natural language processing technology. The text subtitles can be used for display or played together with the multimedia data.

[0060] For example, when the first multimedia data includes video data or image data, the AI ​​processing device uses an image recognition model to identify objects, scenes, text, or features in the image or video data, and generates corresponding text tags. These text tags are used to describe image content, labels, and other information.

[0061] S203, the first device transmits second multimedia data to the second device through the second transmission channel at the second moment.

[0062] The data content of the second multimedia data is the same as that of the first multimedia data. The second moment is later than the first moment, and the interval between the two moments is a preset duration, which is related to the time taken for the second device to complete the AI ​​processing.

[0063] The data content of the second multimedia data is the same as that of the first multimedia data, which means that the audio data, video data and image data in the second multimedia data are completely consistent with those in the first multimedia data and will not change in content due to different transmission channels and transmission times.

[0064] The preset duration is related to the time taken for the second device to complete AI processing. For example, the preset duration is based on the time taken for the second device to complete AI processing, i.e., the time taken for the AI ​​processing device to process multimedia data. For instance, the preset duration is the time taken for the second device to complete AI processing. Another example is that the preset duration is the sum of the time taken for the AI ​​processing device to process multimedia data and a time margin. For example, the preset duration is 4 seconds, the time taken for the AI ​​processing device to process multimedia data is 3.5 seconds, and the time margin is 0.5 seconds.

[0065] The second transmission channel is used to transmit multimedia data for display or playback to the second device.

[0066] S204. The second device aligns the second multimedia data and the processed multimedia data, and outputs the second multimedia data and the processed multimedia data synchronously.

[0067] In one example, the second device aligns the second multimedia data and the processed multimedia data, for example, by matching the playback time corresponding to the processed multimedia data with the playback time corresponding to the second multimedia data based on time stamp information representing the acquisition time; that is, using an absolute time alignment method. In another example, the second device aligns the second multimedia data and the processed multimedia data, for example, by matching the playback time corresponding to the processed multimedia data with the playback time corresponding to the second multimedia data based on the time difference between a preset duration and the actual processing time of the AI; that is, using a relative time alignment method.

[0068] Synchronous output of the second multimedia data and the processed multimedia data refers to maintaining temporal consistency between the two. For example, while playing audio or video data from the second multimedia data, corresponding text subtitles are displayed simultaneously. Alternatively, while displaying image data from the second multimedia data, corresponding text markers are simultaneously overlaid or displayed sideways. Or, the second multimedia data and the processed multimedia data are encapsulated into a synchronized data stream.

[0069] Based on the above, it can be understood that the first device transmits the first multimedia data to the second device in advance through the first transmission channel, ensuring that the AI ​​processing device can perform AI processing in advance. The first device transmits the second multimedia data to the second device after a preset time delay through the second transmission channel, ensuring that AI processing is completed when the second multimedia data is received, achieving time matching between the second multimedia data and the processed multimedia data. Thus, through a simple dual-channel time-division transmission mechanism, audio and video synchronization in high-latency scenarios is achieved.

[0070] In this embodiment, the first device transmits first multimedia data to the second device through a first transmission channel at a first moment; the second device performs AI processing on the first multimedia data through an AI processing device to obtain processed multimedia data; the first device transmits second multimedia data to the second device through a second transmission channel at a second moment, the data content of the second multimedia data being the same as the data content of the first multimedia data, the second moment being later than the first moment, and the time interval between the second moment and the first moment being a preset duration, the preset duration being related to the time taken for the second device to complete the AI ​​processing; the second device aligns the second multimedia data and the processed multimedia data, and synchronously outputs the second multimedia data and the processed multimedia data.

[0071] Based on this, the first device transmits the first multimedia data to the second device through the first transmission channel at the first moment, enabling the second device to receive the first multimedia data in advance and perform AI processing. By introducing a secondary transmission mechanism with a preset duration for the time required for the second device to complete AI processing, the second multimedia data, which was originally transmitted synchronously with the first multimedia data, is converted into a data stream transmitted with a preset delay. This allows the multimedia synchronization system to support latency adaptation within a second or even longer range, matching the latency characteristics of the AI ​​processing link. Furthermore, after receiving the second multimedia data, the second device aligns the timing-matched second multimedia data with the processed multimedia data and outputs them synchronously, ensuring that the second multimedia data and the processed multimedia data are consistent in time. Thus, the problem of audio and video asynchrony in AI multimedia processing application scenarios can be effectively solved.

[0072] In some embodiments, Figure 3 This is another flowchart illustrating the synchronization method provided in an embodiment of this application. Figure 3 As shown, the implementation process of S204 above may include: S301, the second device acquires the first time identifier information of the first multimedia data and the second time identifier information of the second multimedia data respectively.

[0073] The first time identifier is used to characterize the first acquisition time of the first multimedia data. The second time identifier is used to characterize the second acquisition time of the second multimedia data. For example, both the first and second time identifiers are timestamps.

[0074] S302, the second device aligns the second multimedia data and the processed multimedia data according to the first time identifier information and the second time identifier information, and synchronously outputs the second multimedia data and the processed multimedia data.

[0075] The second device uses the first time identifier information and the second time identifier information as time references, and determines the temporal correspondence between the second multimedia data and the processed multimedia data through the matching relationship of the time identifier information, so as to complete the time alignment of the second multimedia data and the processed multimedia data.

[0076] In this embodiment, the second device acquires first time identifier information of the first multimedia data and second time identifier information corresponding to the second multimedia data. Based on this, the second device aligns the second multimedia data with the processed multimedia data using the first and second time identifier information as time references, ensuring that the second multimedia data and the processed multimedia data are time-consistent. This achieves synchronized output of the second multimedia data and the processed multimedia data. By using time identifier information for alignment calibration, timing offsets caused by transmission delay fluctuations, processing time differences, and other factors can be avoided, improving the accuracy and stability of audio and video synchronization in AI multimedia processing scenarios.

[0077] In some embodiments, the implementation process of S302 described above may include: if the first acquisition time represented by the first time identifier information is greater than the second acquisition time represented by the second time identifier information, after waiting for a first duration, the second device outputs the second multimedia data and the processed multimedia data. If the first acquisition time represented by the first time identifier information is equal to the second acquisition time represented by the second time identifier information, the second device outputs the second multimedia data and the processed multimedia data in real time.

[0078] The first duration is the interval between the first and second data acquisition times.

[0079] Specifically, when the first acquisition time is greater than the second acquisition time, it means that the second device receives the processed multimedia data earlier than the time it receives the second multimedia data. When the first time identifier information is equal to the second time identifier information, it means that the second device receives the processed multimedia data at the same time as the time it receives the second multimedia data.

[0080] In this embodiment, when the first acquisition time represented by the first time identifier information is greater than the second acquisition time represented by the second time identifier information, the second device waits for a first duration before outputting the second multimedia data and the processed multimedia data; when the first acquisition time represented by the first time identifier information is equal to the second acquisition time represented by the second time identifier information, the second multimedia data and the processed multimedia data are output in real time. Therefore, based on the comparison result of the first and second time identifier information, differentiated output control of the second multimedia data and the processed multimedia data is achieved, ensuring that the second multimedia data and the processed multimedia data remain consistent in time, enabling the processed multimedia data to be aligned with the second multimedia data.

[0081] In some embodiments, the synchronization method further includes: when the first acquisition time represented by the first time identifier information is less than the second acquisition time represented by the second time identifier information, the second device performs a target operation on the target multimedia data.

[0082] The target operation includes discarding the target multimedia data, which is the processed multimedia data corresponding to the first-time identification information.

[0083] For example, besides discarding the target multimedia data, the target operation can also be: pausing playback of any multimedia data, i.e., neither the target multimedia data nor the second multimedia data is output. During the pause, the second device transmits a retransmission command for the second transmission channel to the first device. This retransmission command instructs the retransmission of multimedia data corresponding to the first time identifier information corresponding to the target multimedia data. For example, if the first time identifier information corresponding to the target multimedia data is data from acquisition time 1 to acquisition time 5, then the retransmission command instructs the retransmission of data from acquisition time 1 to acquisition time 5 on the second transmission channel. After receiving the retransmission command, the first device retransmits the data indicated by the command through the second transmission channel. As an example, the first device can expand the capacity of its buffer device to store the data indicated by the retransmission command for transmission. After receiving the retransmitted data, the second device realigns the target multimedia data and the second multimedia data. Once the target multimedia data and the second multimedia data are aligned, the second device synchronously outputs the target multimedia data and the second multimedia data.

[0084] Wherein, when the first acquisition time represented by the first time identifier information is less than the second acquisition time represented by the second time identifier information, it means that the time when the second device receives the processed multimedia data is later than the time when it receives the second multimedia data.

[0085] In one example, please refer to 4. Figure 4 This diagram illustrates the relationship between the preset duration and the first and second moments provided in the embodiments of this application. Figure 4 As shown, if the preset duration T between the first time t1 and the second time t2 was too short in the previous instance, resulting in the first acquisition time represented by the first time identifier information being less than the second acquisition time represented by the second time identifier information, thus preventing the second device from aligning the second multimedia data and the processed media data, then in the current instance, the second device transmits the second multimedia data to the second device through the second transmission channel at the third time t3. By using the third time t3, which is later than the second time t2, an extended preset duration T' is obtained, enabling the second device to align the second multimedia data and the processed media data.

[0086] For example, the first device typically divides the first multimedia data into multiple data blocks and transmits them to the second device in batches.

[0087] In some embodiments, Figure 5 This is another schematic flowchart illustrating the synchronization method provided in an embodiment of this application. For example... Figure 5 As shown, the implementation process of S204 above may include: S401, The second device obtains the processing time of the AI ​​processing device in performing AI processing on the first multimedia data.

[0088] For example, the second device obtains the completion timestamp of the processed multimedia data and determines the processing duration as the time difference between the completion timestamp and the first moment.

[0089] S402. The second device aligns the second multimedia data and the processed multimedia data according to the processing time and the preset time, and outputs the second multimedia data and the processed multimedia data synchronously.

[0090] Specifically, by using the difference between the processing time and the preset time as a time compensation amount, the processed multimedia data is output with a delay based on this time compensation amount, so that the processed multimedia data and the second multimedia data are consistent in time.

[0091] In this embodiment, the second device acquires the processing time of the AI ​​processing device on the first multimedia data and combines it with a preset time to perform time alignment on the second multimedia data and the processed multimedia data, thereby achieving synchronous output of the second multimedia data and the processed multimedia data. By combining the processing time and the preset time for time alignment, the second device can correct the timing deviation caused by long AI processing in real time, ensuring that the second multimedia data and the processed multimedia data are precisely aligned on the timeline. This avoids over-buffering or data race-out caused by fixed delay strategies, significantly improving the accuracy and stability of audio and video synchronization in AI multimedia processing scenarios.

[0092] In some embodiments, the implementation process of S402 above may include: when the processing time is equal to the preset time, outputting the second multimedia data and the processed multimedia data in real time; when the processing time is less than the preset time, obtaining the third moment when the AI ​​processing is completed, and outputting the second multimedia data and the processed multimedia data after the second time has elapsed from the third moment.

[0093] The second duration is the difference between the preset duration and the processing duration.

[0094] Specifically, when the processing time equals the preset time, it means that the second device receives the processed multimedia data at the same time as it receives the second multimedia data. When the processing time is less than the preset time, it means that the second device receives the processed multimedia data earlier than it receives the second multimedia data.

[0095] In this embodiment, when the processing time equals a preset time, the second multimedia data and the processed multimedia data are output in real time. When the processing time is less than the preset time, the third moment after the completion of AI processing is obtained, and from the third moment, after a second time period, the second multimedia data and the processed multimedia data are output. Therefore, based on the comparison between the processing time and the preset time, differentiated output control of the second multimedia data and the processed multimedia data is achieved, ensuring that the second multimedia data and the processed multimedia data remain consistent in time, allowing the processed multimedia data to be aligned with the second multimedia data.

[0096] In some embodiments, the synchronization method further includes: if the processing time exceeds a preset time, the second device discards the target multimedia data.

[0097] The target multimedia data refers to the multimedia data corresponding to the processing time.

[0098] When the processing time exceeds the preset time, it means that the second device receives the processed multimedia data later than it receives the second multimedia data.

[0099] In some embodiments, the first device further includes a data acquisition device and a caching device. The first multimedia data is data acquired by the first device through the data acquisition device and stored in the caching device. After the second device discards the target multimedia data, the synchronization method further includes: the second device transmitting a control command to the first device; and in response to the control command, the first device expanding the capacity of the caching device.

[0100] Among them, the control instructions are used to instruct the cache space of the cache device to be increased.

[0101] For example, increasing the cache space of a caching device can be achieved by using a portion of the memory of the first device as a cache, such as... Figure 6 As shown, a portion of memory space is used as cache space, thus increasing the cache space.

[0102] For example, increasing the cache space of a cache device can be achieved by adding an external cache device, the size of which can be determined based on the value of the cache space that needs to be expanded.

[0103] In this embodiment, after the second device discards the target multimedia data, it transmits a control command to the first device. In response to the control command, the first device expands the capacity of its buffer device, increasing its buffer space. Therefore, the second device can achieve time synchronization between the second multimedia data and the processed multimedia data during subsequent data processing, avoiding the discarding of the target multimedia data and ensuring the integrity of the multimedia data content played later.

[0104] In some embodiments, the first device further includes an image display device, and the second device further includes an audio playback device. After S204 above, the synchronization method further includes: the second device obtaining target image data and target audio data based on the aligned second multimedia data and the processed multimedia data; the second device transmitting the target image data to the first device; the first device displaying the target image data through the image display device; and the second device playing the target audio data through the audio playback device.

[0105] As an example, between the first device displaying target image data via an image display device and the second device playing target audio data via an audio playback device, the first and second devices negotiate a maximum delay duration through a handshake. Both the first and second devices are aligned according to the maximum delay duration to ensure that the target image data and target audio data are output synchronously. The maximum delay duration is the time required for the first device to perform, for example, data augmentation processing on the target image data after receiving it.

[0106] In this embodiment, the second device obtains target image data and target audio data based on the aligned second multimedia data and the processed multimedia data. The second device transmits the target image data to the first device. The first device displays the target image data via an image display device, and the second device plays the target audio data via an audio playback device. Through the aforementioned echo mechanism, the first device displays the target image data via the image display device, and the second device plays the target audio data via the audio playback device, achieving precise synchronization of the target image data and target audio data on the time axis. This ensures reliable audio and video synchronization even in scenarios with significant transmission or processing delays, effectively improving the adaptability and flexibility of the multimedia synchronization system to meet the multimedia synchronization needs of complex AI application environments.

[0107] Furthermore, before the first device transmits the first multimedia data through the first transmission channel and the second multimedia data through the second transmission channel, the configuration of multimedia channel parameters and the establishment of the first and second transmission channels provide a foundation for subsequent dual-channel data transmission, AI processing, and timing alignment.

[0108] In some examples, the first device configures the channel parameters for the first and second transmission channels, respectively. The first device also configures the timestamp alignment parameters for the first and second transmission channels.

[0109] The timestamp is based on the internal clock of the AI ​​processing device, with a precision of milliseconds.

[0110] For example, the first device configures the buffer range of the first transmission channel and sets the buffer water mark parameters as follows: low water mark for example 0.5 seconds, low middle water mark for example 1 second, middle water mark for example 2 seconds, high middle water mark for example 3 seconds, and high water mark for example 5 seconds.

[0111] For example, the first device configures the buffer range of the second transmission channel and sets the buffer water mark parameters as follows: low water mark for example 0.1 seconds, low middle water mark for example 0.2 seconds, middle water mark for example 0.5 seconds, high middle water mark for example 1 second, and high water mark for example 2 seconds.

[0112] In some examples, the first device establishes a first transmission channel and a second transmission channel, and configures the caching strategies corresponding to the first transmission channel and the second transmission channel.

[0113] Specifically, based on the channel parameters of the first and second transmission channels, a first transmission channel and a second transmission channel are established between the first device and the second device respectively, forming a dual-channel transmission link.

[0114] The first transmission channel uses a Least Recently Used (LRU) caching strategy. The LRU strategy optimizes resource allocation in the multimedia synchronization system, ensuring that the first transmission channel prioritizes the transmission of multimedia data required by AI applications while reserving sufficient resources for other applications, thus avoiding memory leaks caused by cached data accumulation.

[0115] The second transmission channel uses a first-in, first-out (FIFO) caching strategy.

[0116] Through the above, the basic deployment of the multimedia synchronization system is completed, providing stable channel support and parameter basis for subsequent operations such as time-sharing transmission of first and second multimedia data by the first device, and AI processing and timing alignment by the second device.

[0117] It should be understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0118] Based on the foregoing embodiments, this application provides a multimedia synchronization system, which includes a first device and a second device. The second device includes an AI processing unit for performing AI processing on multimedia data. The AI ​​processing includes translating audio data into text subtitles or recognizing image data to generate text tags. The first device and the second device are connected through a first transmission channel and a second transmission channel.

[0119] The first device is used to transmit first multimedia data to the second device through the first transmission channel at a first moment.

[0120] The second device is used to perform AI processing on the first multimedia data through an AI processing unit to obtain processed multimedia data.

[0121] The first device is also used to transmit second multimedia data to the second device through the second transmission channel at the second time. The data content of the second multimedia data is the same as the data content of the first multimedia data. The second time is later than the first time and is separated from the first time by a preset time. The preset time is related to the time taken for the second device to complete the AI ​​processing.

[0122] The second device is also used to align the second multimedia data and the processed multimedia data, and synchronously output the second multimedia data and the processed multimedia data.

[0123] The description of the above multimedia synchronization system embodiments is similar to the description of the above synchronization method embodiments, and has similar beneficial effects. For technical details not disclosed in the multimedia synchronization system embodiments of this application, please refer to the description of the synchronization method embodiments of this application for understanding.

[0124] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the synchronization method provided in the above embodiments.

[0125] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the steps in the method provided in the above-described synchronization method embodiments.

[0126] It should be noted that the descriptions of the storage medium embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0127] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0128] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0129] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0130] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0131] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for synchronizing multimedia data, characterized in that, An application is made in a multimedia synchronization system, the multimedia synchronization system including a first device and a second device, the second device including an AI processing device for AI processing of multimedia data, the AI ​​processing including translating audio data into text subtitles or recognizing image data to generate text tags, the first device and the second device being connected via a first transmission channel and a second transmission channel, the synchronization method including: The first device transmits first multimedia data to the second device through the first transmission channel at a first moment; The second device performs AI processing on the first multimedia data through the AI ​​processing unit to obtain processed multimedia data; The first device transmits second multimedia data to the second device through the second transmission channel at a second time. The data content of the second multimedia data is the same as the data content of the first multimedia data. The second time is later than the first time and is separated from the first time by a preset time. The preset time is related to the time taken for the second device to complete the AI ​​processing. The second device aligns the second multimedia data and the processed multimedia data, and outputs the second multimedia data and the processed multimedia data synchronously.

2. The synchronization method according to claim 1, characterized in that, The second device aligns the second multimedia data and the processed multimedia data, and synchronously outputs the second multimedia data and the processed multimedia data, including: The second device acquires the first time identifier information of the first multimedia data and the second time identifier information of the second multimedia data respectively; the first time identifier information is used to characterize the first acquisition time of the first multimedia data, and the second time identifier information is used to characterize the second acquisition time of the second multimedia data. Based on the first time identifier information and the second time identifier information, the second multimedia data and the processed multimedia data are aligned, and the second multimedia data and the processed multimedia data are output synchronously.

3. The synchronization method according to claim 2, characterized in that, The step of aligning the second multimedia data and the processed multimedia data according to the first time identifier information and the second time identifier information, and synchronously outputting the second multimedia data and the processed multimedia data, includes: If the first acquisition time represented by the first time identifier information is greater than the second acquisition time represented by the second time identifier information, after waiting for a first duration, the second multimedia data and the processed multimedia data are output. The first duration is the interval between the first acquisition time and the second acquisition time. When the first acquisition time represented by the first time identifier information is equal to the second acquisition time represented by the second time identifier information, the second multimedia data and the processed multimedia data are output in real time.

4. The synchronization method according to claim 2, characterized in that, The synchronization method further includes: If the first acquisition time represented by the first time identifier information is less than the second acquisition time represented by the second time identifier information, the second device performs a target operation on the target multimedia data. The target operation includes discarding the target multimedia data, which is the processed multimedia data corresponding to the first time identifier information.

5. The synchronization method according to claim 1, characterized in that, The second device aligns the second multimedia data and the processed multimedia data, and synchronously outputs the second multimedia data and the processed multimedia data, including: The second device obtains the processing time of the AI ​​processing device in performing AI processing on the first multimedia data; Based on the processing time and the preset time, the second multimedia data and the processed multimedia data are aligned and output synchronously.

6. The synchronization method according to claim 5, characterized in that, The step of aligning the second multimedia data and the processed multimedia data according to the processing time and the preset time, and synchronously outputting the second multimedia data and the processed multimedia data, includes: When the processing time is equal to the preset time, the second multimedia data and the processed multimedia data are output in real time. If the processing time is less than the preset time, the third moment when the AI ​​processing is completed is obtained, and after a second time from the third moment, the second multimedia data and the processed multimedia data are output, where the second time is the difference between the preset time and the processing time.

7. The synchronization method according to claim 5, characterized in that, The synchronization method further includes: If the processing time exceeds the preset time, the second device performs a target operation on the target multimedia data, the target operation including discarding the target multimedia data, the target multimedia data being the multimedia data corresponding to the processing time.

8. The synchronization method according to claim 4 or 7, characterized in that, The first device further includes a data acquisition device and a buffer device. The first multimedia data is data acquired by the first device through the data acquisition device and stored in the buffer device. After the second device performs a target operation on the target multimedia data, the synchronization method further includes: The second device transmits control commands to the first device, the control commands being used to instruct an increase in the cache space of the cache device; In response to the control command, the first device expands the capacity of the cache device.

9. The synchronization method according to any one of claims 1-7, characterized in that, The first device further includes an image display device, and the second device further includes an audio playback device. After the second device aligns the second multimedia data and the processed multimedia data and synchronously outputs the second multimedia data and the processed multimedia data, the synchronization method further includes: The second device obtains target image data and target audio data based on the aligned second multimedia data and the processed multimedia data; The second device transmits the target image data to the first device; The first device displays the target image data through the image display device, and the second device plays the target audio data through the audio playback device.

10. A multimedia synchronization system, characterized in that, The multimedia synchronization system includes a first device and a second device. The second device includes an AI processing unit for AI processing of multimedia data. The AI ​​processing includes translating audio data into text subtitles or recognizing image data to generate text tags. The first device and the second device are connected through a first transmission channel and a second transmission channel. The first device is configured to transmit first multimedia data to the second device through a first transmission channel at a first moment; The second device is used to perform AI processing on the first multimedia data through the AI ​​processing device to obtain processed multimedia data; The first device is also configured to transmit second multimedia data to the second device through a second transmission channel at a second time. The data content of the second multimedia data is the same as the data content of the first multimedia data. The second time is later than the first time and is separated from the first time by a preset time interval. The preset time interval is related to the time taken for the second device to complete the AI ​​processing. The second device is further configured to align the second multimedia data and the processed multimedia data, and synchronously output the second multimedia data and the processed multimedia data.

Images