Data synchronization method and device, storage medium and electronic equipment

Abstract

The invention discloses a data synchronization method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining an audio frame transmitted by a transmitting end, inserting the audio frame into an audio frame buffer queue, querying a current timestamp of a reference clock, converting the current timestamp into a target media timestamp, and transmitting the target media timestamp to a receiving end. And acquiring a head media timestamp of the audio frame which is ranked at the first in the audio frame buffer queue, and if the target media timestamp is greater than the head media timestamp, deleting the audio frame of which the media timestamp is less than the target media timestamp in the audio frame buffer queue. By adopting the method and the device, the audio frames with relatively long delay time in the audio frame buffer queue are deleted, so that the same audio can be output at the same time point as the sending end, and the data synchronism with the sending end is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method, device, storage medium, and electronic device. Background Art

[0002] AES67 is an open network digital audio standard. It's based on an IP network architecture and utilizes existing IT network protocols to achieve interoperability guidelines for high-performance professional audio transmission. Existing technologies often use the AES67 protocol for audio transmission between devices, thereby achieving audio data synchronization between different devices. This allows different devices to play the same audio data simultaneously. However, due to issues with device hardware and the data transmission environment, audio transmission, rendering, or playback can easily become asynchronous. This can lead to different audio outputs from different devices being out of sync, meaning different audio is output at the same time. Therefore, a data synchronization method is needed to ensure the synchronization of audio outputs from different devices. Summary of the Invention

[0003] The present invention provides a data synchronization method, device, storage medium, and electronic device that can improve data synchronization with the sending end by deleting audio frames with large delays in the audio frame buffer queue, so that the same audio can be output at the same time as the sending end. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a data synchronization method, the method comprising:

[0005] Obtaining the audio frame sent by the sending end, and inserting the audio frame into the audio frame buffer queue;

[0006] Querying a current timestamp of a reference clock, and converting the current timestamp into a target media timestamp;

[0007] Obtaining a header media timestamp of the audio frame that is first sorted in the audio frame buffer queue;

[0008] If the target media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp are deleted.

[0009] In a second aspect, an embodiment of the present application provides a data synchronization device, the device comprising:

[0010] An audio frame receiving module is used to obtain the audio frame sent by the sending end and insert the audio frame into the audio frame buffer queue;

[0011] A timestamp conversion module, configured to query a current timestamp of a reference clock and convert the current timestamp into a target media timestamp;

[0012] A header timestamp acquisition module, configured to acquire a header media timestamp of the audio frame that is first sorted in the audio frame buffer queue;

[0013] The audio frame deletion module is configured to delete the audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp if the target media timestamp is greater than the header media timestamp.

[0014] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0016] In one or more embodiments of the present application, the audio frame sent by the transmitter is obtained, the audio frame is inserted into the audio frame buffer queue, the current timestamp of the reference clock is queried, the current timestamp is converted into a target media timestamp, the header media timestamp of the audio frame sorted first in the audio frame buffer queue is obtained, and if the target media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp are deleted. By deleting the audio frames with a larger delay time in the audio frame buffer queue, the same audio can be output at the same time point as the transmitter, thereby improving data synchronization with the transmitter. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is an example diagram of data synchronization provided by an embodiment of the present application;

[0019] Figure 2 This is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0020] Figure 3 This is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0021] Figure 4This is an example schematic diagram of blank frame insertion provided in an embodiment of the present application;

[0022] Figure 5 This is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0023] Figure 6 This is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0024] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] The transmitter can exchange data with the receiver based on the AES67 protocol. The transmitter can send audio data frame by frame to the receiver. After receiving the audio frames, the receiver can insert the audio frames into the audio frame buffer queue and play the audio frames sequentially according to the order in the audio frame buffer queue to achieve the purpose of synchronizing the audio data between the transmitter and the receiver. It is understood that the transmitter and the receiver can transmit audio data to each other based on the AES67 protocol. The transmitter can be connected to multiple receivers, and the receiver can also be connected to multiple transmitters. In addition to sending audio data, the transmitter can also receive audio data. Similarly, the receiver can also send audio data in addition to receiving audio data. The transmitter and receiver can be terminal devices such as mobile phones, computers, tablets, and wearable devices. The data synchronization device can be the receiver or a module in the receiver that implements the data synchronization method.

[0027] Please also see Figure 1, an example schematic diagram of data synchronization is provided for an embodiment of the present application. The data synchronization device can receive the audio frame sent by the sending end, and the sending end can send audio frame 4 to the receiving end. After the data synchronization device obtains the audio frame 4 sent by the sending end, the audio frame 4 can be inserted into the end of the audio frame buffer queue. It can be understood that the other audio frames in the audio frame buffer queue are audio frames received before audio frame 4 and have not yet been played, such as audio frame 1, audio frame 2 and audio frame 3, which are audio frames received by the receiving end before receiving audio frame 4. After receiving the audio frame, the data synchronization device can also insert the audio frame 4 in the audio frame buffer queue. Audio frames with excessively long delays are deleted. For example, after receiving audio frame 4, the data synchronization device can obtain the current timestamp of the reference clock and convert the current timestamp into the corresponding target media timestamp. If the media timestamp of audio frame 1 is less than the target media timestamp, it means that the sender has played audio frame 1, but the receiver has not yet played audio frame 1, that is, there is data asynchrony between the sender and the receiver. The data synchronization device will delete audio frame 1 and remove audio frame 1 from the audio frame buffer queue. The receiver will continue to play from audio frame 2, thereby synchronizing the data between the receiver and the sender.

[0028] The reference clock can be a standard clock, i.e., the local clock of the terminal device. The local clocks of all senders and all receivers are the same. The timestamp of the reference clock changes naturally over time. The media timestamp is a timestamp for the audio data, used to indicate the position of the audio data played. The media timestamp of an audio frame is based on the media timestamp of the previous audio frame, the frequency of the audio data sent by the sender, and the time interval between adjacent audio frames. For example, if the sender sends audio data at a frequency of 50 frames / s, i.e., 50 audio frames per second, the time interval between adjacent audio frames is 20ms. If the media timestamp of audio frame 1 is 10:00:00, then the media timestamp of audio frame 2 is 10:00:20, the media timestamp of audio frame 3 is 10:00:40, and the media timestamp of audio frame 4 is 10:00:60. The frequency can be the initial setting of the sender or can be set by the user or relevant staff.

[0029] The data synchronization method provided in this application is described in detail below with reference to specific embodiments.

[0030] See Figure 2 , provides a flow chart of a data synchronization method according to an embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps S101-S104.

[0031] S101, obtaining an audio frame sent by a sending end, and inserting the audio frame into an audio frame buffer queue.

[0032] Specifically, the transmitter can send audio data to the receiver at a transmission frequency. That is, the transmitter sends audio frames frame by frame at the transmission frequency. The data synchronization device obtains the audio frames sent by the transmitter and inserts the audio frames into the end of the audio frame buffer queue. The receiver can play the audio frames in the audio frame buffer queue in the order in which the audio frames are arranged.

[0033] It is understood that after receiving the audio frame, the data synchronization device can verify the audio frame and insert the audio frame into the audio frame buffer queue if the audio frame verification is correct. The data synchronization device will simultaneously render the audio frame after inserting it into the audio frame buffer queue, so that the receiving end can play the audio frame later.

[0034] S102: Query the current timestamp of the reference clock, and convert the current timestamp into a target media timestamp.

[0035] Specifically, the data synchronization device can query the current timestamp of the reference clock, that is, the data synchronization device can query the current timestamp corresponding to the reference clock at the time point when the audio frame is received, and then convert the current timestamp into a media timestamp to obtain the target media timestamp corresponding to the current timestamp. It can be understood that since the reference clocks and current timestamps of the receiving and sending ends are the same, the media timestamp corresponding to the current timestamp is the media timestamp of the audio played by the sending end at the current timestamp, indicating the playback progress of the audio on the sending end.

[0036] S103: Obtain the header media timestamp of the audio frame that is first sorted in the audio frame buffer queue.

[0037] Specifically, the data synchronization device can obtain the media timestamp of the audio frame that is first sorted in the audio frame buffer queue and confirm it as the header media timestamp. The header media timestamp is the earliest media timestamp in the audio frame buffer queue. The audio frame that is first sorted in the audio frame buffer queue is the next audio frame to be played by the receiving end.

[0038] S104: If the target media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamp are deleted.

[0039] Specifically, if the target media timestamp is greater than the header media timestamp, it indicates that there is audio asynchrony between the receiving end and the sending end, that is, the audio playback progress of the receiving end lags behind the audio playback progress of the sending end, and there is at least one audio frame in the audio frame buffer queue that has been played by the sending end but not yet played by the receiving end. It can be understood that all audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamps are audio frames that have been played by the sending end but not yet played by the receiving end. If these audio frames continue to remain in the audio frame buffer queue waiting for the receiving end to play them, it will cause obvious audio data asynchrony between the receiving end and the sending end. Therefore, the data synchronization device can delete the audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamps, thereby achieving the purpose of audio data synchronization between the receiving end and the sending end.

[0040] In an embodiment of the present application, the audio frame sent by the transmitter is obtained, the audio frame is inserted into the audio frame buffer queue, the current timestamp of the reference clock is queried, the current timestamp is converted into a target media timestamp, the header media timestamp of the audio frame sorted first in the audio frame buffer queue is obtained, and if the target media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp are deleted. By deleting the audio frames with a larger delay time in the audio frame buffer queue, the same audio can be output at the same time point as the transmitter, thereby improving data synchronization with the transmitter.

[0041] See Figure 3 , provides a flow chart of a data synchronization method according to an embodiment of the present application. Figure 3 As shown, the method of the embodiment of the present application may include the following steps S201-S208.

[0042] S201: Acquire an audio frame sent by a transmitter, and acquire a receiving media timestamp of the audio frame.

[0043] Specifically, the sending end can send audio data to the receiving end according to the sending frequency, that is, the sending end sends audio frames frame by frame according to the sending frequency, and the data synchronization device obtains the audio frames sent by the sending end. It can be understood that after receiving the audio frame, the data synchronization device can verify the audio frame. If the audio frame verification is correct, the audio frame can be inserted into the audio frame buffer queue. The data synchronization device will synchronously render the audio frame when inserting the audio frame into the audio frame buffer queue, so that the receiving end can play the audio frame later. Similarly, if the audio frame verification is wrong, the audio frame can be discarded, and then a blank frame can be inserted into the audio frame buffer queue to fill the position of the audio frame in the audio playback process, thereby ensuring that the audio frame is intact and avoiding audio damage during audio playback.

[0044] S202: If the audio frame buffer queue is not empty, obtain the tail media timestamp of the last audio frame sorted in the audio frame buffer queue.

[0045] Specifically, the data synchronization device can determine whether the audio frame buffer queue is empty. If the audio frame buffer queue is not empty, that is, there are still audio frames in the audio frame buffer queue waiting to be played, then the data synchronization device can obtain the media timestamp corresponding to the last audio frame sorted in the audio frame buffer queue, that is, the tail media timestamp. The audio frame corresponding to the tail media timestamp is the previous audio frame received by the receiving end before the current audio frame.

[0046] Optionally, if the audio frame buffer queue is not empty, that is, there are no audio frames waiting to be played in the audio frame buffer queue, the data synchronization device can directly insert the audio frame into the audio frame buffer queue. It can be understood that after the audio frame is inserted into the audio frame buffer queue, the head media timestamp and tail media timestamp of the audio frame buffer queue are both the receiving media timestamp of the audio frame.

[0047] S203: If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the sending interval, perform blank frame insertion processing on the audio frame buffer queue.

[0048] Specifically, during the data transmission process between the sender and the receiver, audio frame errors, audio frame losses, etc. may occur, so it is necessary to obtain the timestamp interval between the tail media timestamp and the receiving media timestamp, and judge whether there is a frame drop between the tail media timestamp and the receiving media timestamp based on the timestamp interval and the sending interval of the sender. The sending interval is the time interval between two adjacent audio frames sent by the sender. If the timestamp interval is greater than or equal to twice the sending interval, it means that there is a frame drop between the tail media timestamp and the receiving timestamp. If there is a frame drop, blank frames need to be inserted into the audio frame buffer queue to make up for the lost audio frames between the tail media timestamp and the receiving media timestamp.

[0049] Optionally, the data synchronization device can obtain the timestamp interval between the tail media timestamp and the received media timestamp, and obtain the sending frequency of the sending end, calculate the sending interval of the sending end based on the sending frequency, and calculate the number of insertion frames that need to be inserted into the audio frame buffer queue based on the timestamp interval and the sending interval, and then insert blank frames that meet the number of insertion frames at the end of the audio frame buffer queue.

[0050] S204: Insert the audio frame into the audio frame buffer queue.

[0051] Specifically, the data synchronization device may insert the audio frame into the tail of the audio frame buffer queue, and the receiving end may play the audio frames in the audio frame buffer queue in sequence according to the arrangement order of the audio frames in the audio frame buffer queue.

[0052] Please also see Figure 4 , an example schematic diagram of blank frame insertion is provided for the embodiment of the present application. If the data synchronization device obtains the audio frame D received by the receiving end, the receiving media timestamp corresponding to the audio frame D is 10:00:50, and the audio frame buffer queue is not empty, wherein there are audio frames A, audio frame B and audio frame C waiting to be played, and the corresponding media timestamps are 10:00:00, 10:00:10 and 10:00:20 respectively, the last audio frame sorted in the audio frame buffer queue is audio frame C, and the tail media timestamp is 10:00:20. The data synchronization device can obtain the sending The sending frequency of the end, if the sending frequency is 100 frames / s, that is, the sending end sends 100 audio frames per second, then the sending interval should be 10ms, and the tail media timestamp and the receiving media timestamp, that is, the timestamp interval between audio frame C and audio frame D is 30ms, so two audio frames are missing before audio frame D, and the number of inserted frames that need to be inserted with blank frames is 2. The data synchronization device can insert two blank frames into the tail of the audio frame buffer queue, that is, behind audio frame C. After completing the blank frame insertion process, the audio frame D is inserted into the tail of the audio frame buffer queue, that is, behind the blank frame.

[0053] It is understandable that when an audio frame is inserted into the audio frame buffer queue, the tail media timestamp of the audio frame buffer queue changes to the current media timestamp corresponding to the audio frame, such as Figure 4 As shown in FIG, if the audio frame D is inserted into the audio frame buffer queue, the tail media timestamp of the audio frame buffer queue changes from 10:00:20 to 10:00:50.

[0054] S205: Query the current timestamp of the reference clock and obtain the target media timestamp corresponding to the current timestamp.

[0055] Specifically, the data synchronization device can query the current timestamp of the reference clock, that is, the data synchronization device can query the current timestamp corresponding to the reference clock at the time point when the audio frame is received, and then convert the current timestamp into a media timestamp to obtain the target media timestamp corresponding to the current timestamp. It can be understood that since the reference clocks and current timestamps of the receiving and sending ends are the same, the media timestamp corresponding to the current timestamp is the media timestamp of the audio played by the sending end at the current timestamp, indicating the playback progress of the audio on the sending end.

[0056] Optionally, the data synchronization device may calculate the corresponding target media timestamp T2 based on the current timestamp T1 of the reference clock, using the following formula:

[0057] T2=(T1×f)MODMAXOffset

[0058] Wherein, f is the sampling frequency of the receiving end, that is, the sending frequency of the sending end, and MAXOffset is the maximum offset value of the media timestamp, MaxOffset=0×FFFFFFFF.

[0059] S206: Obtain the header media timestamp of the audio frame that is first sorted in the audio frame buffer queue.

[0060] Specifically, the data synchronization device can obtain the media timestamp of the audio frame that is first sorted in the audio frame buffer queue and confirm it as the header media timestamp. The header media timestamp is the earliest media timestamp in the audio frame buffer queue. The audio frame that is first sorted in the audio frame buffer queue is the next audio frame to be played by the receiving end.

[0061] S207: If the target media timestamp is greater than the header media timestamp, obtain a delayed media timestamp corresponding to the target media timestamp.

[0062] Specifically, if the target media timestamp is greater than the header media timestamp, it indicates that there is audio asynchrony between the receiving end and the sending end, that is, the audio playback progress of the receiving end lags behind the audio playback progress of the sending end, and there is at least one audio frame in the audio frame buffer queue that has been played by the sending end but not yet played by the receiving end. It can be understood that all audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamps are audio frames that have been played by the sending end but not yet played by the receiving end. If these audio frames continue to remain in the audio frame buffer queue waiting for the receiving end to play them, it will cause obvious audio data asynchrony between the receiving end and the sending end. Therefore, the data synchronization device can delete the audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamps, thereby achieving the purpose of audio data synchronization between the receiving end and the sending end.

[0063] It is understandable that data transmission requires a certain amount of time, so delays in data transmission between the receiving end and the sending end are inevitable. The data synchronization device can allow a certain buffer delay interval between the audio played by the receiving end and the sending end. The buffer delay interval is the maximum delay time interval between the audio played by the receiving end and the sending end. If the target media timestamp is greater than the header media timestamp, the delayed media timestamp corresponding to the target media timestamp can be obtained. The delayed media timestamp is the media timestamp corresponding to the maximum delay interval allowed by the data synchronization device based on the target media timestamp.

[0064] Optionally, if the target media timestamp is greater than the header media timestamp, the data synchronization device may obtain a buffer delay interval, where the buffer delay interval may be an initial setting of the data synchronization device or may be configured by the user or relevant staff. For example, it may be 4ms, meaning that the maximum delay between the receiving end and the sending end may be 4ms. The data synchronization device may subtract the buffer delay interval from the target media timestamp to obtain a delayed media timestamp corresponding to the target media timestamp. For example, if the target media timestamp is 10:00:00 and the buffer delay interval is 4ms, then the buffer delay timestamp corresponding to the target media timestamp is 10:00:04.

[0065] S208: If the delayed media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are less than the delayed media timestamp are deleted.

[0066] Specifically, if the delayed media timestamp is still greater than the header media timestamp, the data synchronization device may delete the audio frames in the audio frame buffer queue whose media timestamps are greater than the delayed media timestamp. It is understood that if the delayed media timestamp is less than or equal to the header media timestamp, the data synchronization device does not need to delete the audio frames in the audio frame buffer queue.

[0067] Optionally, if the delayed media timestamp is greater than the header media timestamp, the delay interval between the delayed media timestamp and the header media timestamp is obtained. The delay interval is the difference between the delayed media timestamp and the header media timestamp. The data synchronization device can obtain the number of frames to be deleted based on the delay interval. It can be understood that the data synchronization device can calculate the number of frames to be deleted based on the delay interval and the sending frequency of the sender. The number of frames to be deleted is the number of audio frames in the audio frame buffer queue whose media timestamp is less than the delayed media timestamp. The data synchronization device can delete the audio frames in the audio frame buffer queue that meet the data to be deleted in chronological order. The data synchronization device can sort the audio frames in the audio frame buffer queue and delete the audio frames with the number of frames to be deleted before deleting.

[0068] Optionally, the data synchronization device may control the receiving end to play the audio frames in the audio frame buffer queue in sequence according to the order of the audio frames in the audio frame buffer queue.

[0069] In an embodiment of the present application, the audio frame sent by the transmitter is obtained, the receiving media timestamp of the audio frame is obtained, and the data synchronization device can verify the audio frame to ensure that the audio frame is intact and avoid audio damage during audio playback. If the audio frame buffer queue is not empty, the tail media timestamp of the last audio frame sorted in the audio frame buffer queue is obtained, and based on the timestamp interval between the tail media timestamp and the receiving media timestamp, the audio frame buffer queue is inserted with a blank frame. If there is a frame drop, the audio frame buffer queue is inserted with a blank frame to make up for the lost audio frame between the tail media timestamp and the receiving media timestamp, and the audio frame is inserted into the audio frame buffer queue. The current timestamp of the reference clock is queried and the current timestamp is converted into the target media timestamp. If the target media timestamp is greater than the head media timestamp, the delayed media timestamp corresponding to the target media timestamp is obtained. If the delayed media timestamp is still greater than the head media timestamp, the audio frame in the audio frame buffer queue whose media timestamp is less than the delayed media timestamp is deleted. The data synchronization device can control the receiving end to play the audio frames in the audio frame buffer queue in sequence according to the order of the audio frames in the audio frame buffer queue. By deleting audio frames with large delays in the audio frame buffer queue, the receiving end can output the same audio at the same time as the sending end, thereby improving data synchronization with the sending end.

[0070] The following will be combined with the Figure 5 -Attached Figure 6 , the data synchronization device provided by the embodiment of the present application is introduced in detail. Figure 5 -Attached Figure 6 The data synchronization device in the present application is used to execute Figure 2 and Figure 3 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 2 and Figure 3 The embodiment shown.

[0071] See Figure 5 , which shows a schematic diagram of the structure of a data synchronization device provided by an exemplary embodiment of the present application. The data synchronization device can be implemented as all or part of a device through software, hardware, or a combination of both. The device 1 includes an audio frame receiving module 11, a timestamp conversion module 12, a header timestamp acquisition module 13, and an audio frame deletion module 14.

[0072] The audio frame receiving module 11 is used to obtain the audio frame sent by the sending end and insert the audio frame into the audio frame buffer queue;

[0073] A timestamp conversion module 12 is configured to query a current timestamp of a reference clock and convert the current timestamp into a target media timestamp;

[0074] A header timestamp acquisition module 13 is configured to acquire a header media timestamp of the audio frame that is first sorted in the audio frame buffer queue;

[0075] The audio frame deletion module 14 is configured to delete the audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp if the target media timestamp is greater than the header media timestamp.

[0076] In this embodiment, the audio frame sent by the transmitter is obtained, the audio frame is inserted into the audio frame buffer queue, the current timestamp of the reference clock is queried, the current timestamp is converted into a target media timestamp, and the header media timestamp of the audio frame sorted first in the audio frame buffer queue is obtained. If the target media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp are deleted. By deleting the audio frames with a larger delay time in the audio frame buffer queue, the same audio can be output at the same time point as the transmitter, thereby improving data synchronization with the transmitter.

[0077] See Figure 6, which shows a schematic diagram of the structure of a data synchronization device provided by an exemplary embodiment of the present application. The data synchronization device can be implemented as all or part of the device through software, hardware, or a combination of both. It includes an audio frame receiving module 11, a timestamp conversion module 12, a header timestamp acquisition module 13, an audio frame deletion module 14, and an audio frame playback module 15.

[0078] The audio frame receiving module 11 is used to obtain the audio frame sent by the sending end and insert the audio frame into the audio frame buffer queue;

[0079] Optionally, the audio frame receiving module 11 is specifically configured to obtain the audio frame sent by the sending end and obtain a receiving media timestamp of the audio frame;

[0080] If the audio frame buffer queue is not empty, obtaining the tail media timestamp of the last audio frame sorted in the audio frame buffer queue;

[0081] If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the sending interval, performing blank frame insertion processing on the audio frame buffer queue;

[0082] Inserting the audio frame into the audio frame buffer queue.

[0083] Optionally, the audio frame receiving module 11 is specifically configured to obtain a timestamp interval between the tail media timestamp and the received media timestamp, and obtain a sending frequency of the sending end;

[0084] Calculating a sending interval of the sending end based on the sending frequency;

[0085] If the timestamp interval is greater than or equal to twice the sending interval, obtaining the number of inserted frames based on the timestamp interval and the sending interval;

[0086] Insert blank frames that meet the number of inserted frames at the end of the audio frame buffer queue.

[0087] A timestamp conversion module 12 is configured to query a current timestamp of a reference clock and convert the current timestamp into a target media timestamp;

[0088] A header timestamp acquisition module 13 is configured to acquire a header media timestamp of the audio frame that is first sorted in the audio frame buffer queue;

[0089] an audio frame deletion module 14, configured to delete the audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp if the target media timestamp is greater than the header media timestamp;

[0090] Optionally, the audio frame deletion module 14 is specifically configured to obtain a delayed media timestamp corresponding to the target media timestamp if the target media timestamp is greater than the header media timestamp;

[0091] If the delayed media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are greater than the delayed media timestamp are deleted.

[0092] Optionally, the audio frame deletion module 14 is specifically configured to obtain a buffering delay interval if the target media timestamp is greater than the header media timestamp;

[0093] The buffering delay interval is subtracted from the target media timestamp to obtain a delayed media timestamp corresponding to the target media timestamp.

[0094] Optionally, the audio frame deletion module 14 is specifically configured to obtain a delay interval between the delayed media timestamp and the header media timestamp if the delayed media timestamp is greater than the header media timestamp;

[0095] The number of frames to be deleted is obtained based on the delay interval, and audio frames in the audio frame buffer queue that meet the number of frames to be deleted are deleted in chronological order.

[0096] The audio frame playing module 15 is configured to play the audio frames in the audio frame buffer queue in sequence according to the order of the audio frames in the audio frame buffer queue.

[0097] In this embodiment, the audio frame sent by the sender is obtained, the receiving media timestamp of the audio frame is obtained, and the data synchronization device can verify the audio frame to ensure that the audio frame is intact and avoid audio damage during audio playback. If the audio frame buffer queue is not empty, the tail media timestamp of the last audio frame in the audio frame buffer queue is obtained, and based on the timestamp interval between the tail media timestamp and the receiving media timestamp, the audio frame buffer queue is inserted into the blank frame. If there is a frame drop, the audio frame buffer queue is inserted into the blank frame to make up for the lost audio frame between the tail media timestamp and the receiving media timestamp. The audio frame is inserted into the audio frame buffer queue, the current timestamp of the reference clock is queried, and the current timestamp is converted to the target media timestamp. If the target media timestamp is greater than the head media timestamp, the delayed media timestamp corresponding to the target media timestamp is obtained. If the delayed media timestamp is still greater than the head media timestamp, the audio frame in the audio frame buffer queue whose media timestamp is less than the delayed media timestamp is deleted. The data synchronization device can control the receiving end to play the audio frames in the audio frame buffer queue in sequence according to the order of the audio frames in the audio frame buffer queue. By deleting audio frames with large delays in the audio frame buffer queue, the receiving end can output the same audio at the same time as the sending end, thereby improving data synchronization with the sending end.

[0098] It should be noted that the data synchronization device provided in the above embodiment only uses the division of the above functional modules as an example when executing the data synchronization method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data synchronization device provided in the above embodiment and the data synchronization method embodiment are based on the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0099] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0100] The present application also provides a computer storage medium that can store multiple instructions, which are suitable for being loaded and executed by a processor as described above. Figures 1-4 The data synchronization method of the embodiment shown in the figure can be found in the specific execution process. Figures 1-4 The detailed description of the illustrated embodiment will not be repeated here.

[0101] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figures 1-4 The data synchronization method of the embodiment shown in the figure can be found in the specific execution process. Figures 1-4 The detailed description of the illustrated embodiment will not be repeated here.

[0102] Please refer to Figure 7 , which shows a block diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device in the present application may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.

[0103] The processor 110 may include one or more processing cores. The processor 110 utilizes various interfaces and circuits to connect various components within the electronic device. It executes instructions, programs, code sets, or instruction sets stored in the memory 120, as well as accesses data stored in the memory 120, to perform various functions and process data for the terminal 100. Optionally, the processor 110 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 110 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interfaces, and applications; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 110 and may be implemented separately via a communication chip.

[0104] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an iOS system developed by Apple, including a system deeply developed based on the iOS system or other systems.

[0105] The memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve better operating results, the operating system allocates corresponding system resources to different third-party applications. However, the requirements for system resources in different application scenarios in the same third-party application are also different. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and the third-party application are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0106] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0107] The input device 130 is used to receive input commands or data and includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch-sensitive device. The output device 140 is used to output commands or data and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 may be combined, and the input device 130 and the output device 140 may be a touch-sensitive display.

[0108] The touch display screen can be designed as a full screen, a curved screen or a special-shaped screen. The touch display screen can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the present embodiment.

[0109] In addition, those skilled in the art will understand that the structures of the electronic devices shown in the above figures do not limit the electronic devices. The electronic devices may include more or fewer components than shown, or may combine certain components, or arrange the components differently. For example, the electronic devices may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, and other components, which will not be described in detail here.

[0110] exist Figure 7 In the electronic device shown, the processor 110 may be configured to call a data synchronization application stored in the memory 120 and specifically perform the following operations:

[0111] Obtaining the audio frame sent by the sending end, and inserting the audio frame into the audio frame buffer queue;

[0112] Querying a current timestamp of a reference clock, and converting the current timestamp into a target media timestamp;

[0113] Obtaining a header media timestamp of the audio frame that is first sorted in the audio frame buffer queue;

[0114] If the target media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamp are deleted.

[0115] In one embodiment, when the processor 110 acquires the audio frame sent by the sending end and inserts the audio frame into the audio frame buffer queue, the processor 110 specifically performs the following operations:

[0116] Obtaining an audio frame sent by the transmitting end, and obtaining a receiving media timestamp of the audio frame;

[0117] If the audio frame buffer queue is not empty, obtaining the tail media timestamp of the last audio frame sorted in the audio frame buffer queue;

[0118] If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the sending interval, performing blank frame insertion processing on the audio frame buffer queue;

[0119] Inserting the audio frame into the audio frame buffer queue.

[0120] In one embodiment, when performing blank frame insertion processing on the audio frame buffer based on the timestamp interval between the tail media timestamp and the received media timestamp, the processor 110 specifically performs the following operations:

[0121] Obtaining a timestamp interval between the tail media timestamp and the received media timestamp, and obtaining a sending frequency of the sending end;

[0122] Calculating a sending interval of the sending end based on the sending frequency;

[0123] If the timestamp interval is greater than or equal to twice the sending interval, obtaining the number of inserted frames based on the timestamp interval and the sending interval;

[0124] Insert blank frames that meet the number of inserted frames at the end of the audio frame buffer queue.

[0125] In one embodiment, when the processor 110 deletes the audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamp if the target media timestamp is greater than the header media timestamp, the processor 110 specifically performs the following operations:

[0126] If the target media timestamp is greater than the header media timestamp, obtaining a delayed media timestamp corresponding to the target media timestamp;

[0127] If the delayed media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are greater than the delayed media timestamp are deleted.

[0128] In one embodiment, when executing the process of obtaining a delayed media timestamp corresponding to the target media timestamp if the target media timestamp is greater than the header media timestamp, the processor 110 specifically performs the following operations:

[0129] If the target media timestamp is greater than the header media timestamp, obtaining a buffering delay interval;

[0130] The buffering delay interval is subtracted from the target media timestamp to obtain a delayed media timestamp corresponding to the target media timestamp.

[0131] In one embodiment, when the processor 110 executes the process of deleting the audio frames in the audio frame buffer queue whose media timestamps are greater than the delayed media timestamp if the delayed media timestamp is greater than the header media timestamp, the processor 110 specifically performs the following operations:

[0132] If the delayed media timestamp is greater than the header media timestamp, obtaining a delay interval between the delayed media timestamp and the header media timestamp;

[0133] The number of frames to be deleted is obtained based on the delay interval, and audio frames in the audio frame buffer queue that meet the number of frames to be deleted are deleted in chronological order.

[0134] In one embodiment, when executing the data synchronization method, the processor 110 further performs the following operations:

[0135] The audio frames in the audio frame buffer queue are played in sequence according to the order of the audio frames in the audio frame buffer queue.

[0136] In this embodiment, the audio frame sent by the sender is obtained, the receiving media timestamp of the audio frame is obtained, and the data synchronization device can verify the audio frame to ensure that the audio frame is intact and avoid audio damage during audio playback. If the audio frame buffer queue is not empty, the tail media timestamp of the last audio frame in the audio frame buffer queue is obtained, and based on the timestamp interval between the tail media timestamp and the receiving media timestamp, the audio frame buffer queue is inserted into the blank frame. If there is a frame drop, the audio frame buffer queue is inserted into the blank frame to make up for the lost audio frame between the tail media timestamp and the receiving media timestamp. The audio frame is inserted into the audio frame buffer queue, the current timestamp of the reference clock is queried, and the current timestamp is converted to the target media timestamp. If the target media timestamp is greater than the head media timestamp, the delayed media timestamp corresponding to the target media timestamp is obtained. If the delayed media timestamp is still greater than the head media timestamp, the audio frame in the audio frame buffer queue whose media timestamp is less than the delayed media timestamp is deleted. The data synchronization device can control the receiving end to play the audio frames in the audio frame buffer queue in sequence according to the order of the audio frames in the audio frame buffer queue. By deleting audio frames with large delays in the audio frame buffer queue, the receiving end can output the same audio at the same time as the sending end, thereby improving data synchronization with the sending end.

[0137] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0138] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

[0139] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, and display, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the audio frames and current timestamps of the reference clock involved in this specification are all obtained with full authorization.

Claims

1. A data synchronization method, It is characterized in that The method comprises: Obtaining an audio frame sent by a sending end, and inserting the audio frame into an audio frame buffer queue; Querying a current timestamp of a reference clock, and converting the current timestamp into a target media timestamp; Obtaining a header media timestamp of the audio frame that is first sorted in the audio frame buffer queue; If the target media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamp are deleted.

2. The method according to claim 1, It is characterized in that The step of obtaining an audio frame sent by a transmitting end and inserting the audio frame into an audio frame buffer queue comprises: Obtaining the audio frame sent by the sender, and obtaining the receiving media timestamp of the audio frame; If the audio frame buffer queue is not empty, obtaining the tail media timestamp of the last audio frame sorted in the audio frame buffer queue; If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the sending interval, a blank frame insertion process is performed on the audio frame buffer queue; Inserting the audio frame into the audio frame buffer queue.

3. The method according to claim 2, It is characterized in that If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the sending interval, performing blank frame insertion processing on the audio frame buffer queue includes: Obtaining a timestamp interval between the tail media timestamp and the received media timestamp, and obtaining a sending frequency of the sending end; Based on the sending frequency, calculating the sending interval of the sending end; If the timestamp interval is greater than or equal to twice the sending interval, obtaining the number of inserted frames based on the timestamp interval and the sending interval; Insert blank frames that meet the number of inserted frames at the end of the audio frame buffer queue.

4. The method according to claim 1, It is characterized in that If the target media timestamp is greater than the header media timestamp, deleting the audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamp, comprises: If the target media timestamp is greater than the header media timestamp, obtaining a delayed media timestamp corresponding to the target media timestamp; If the delayed media timestamp is greater than the header media timestamp, the audio frames in the audio frame buffer queue whose media timestamps are greater than the delayed media timestamp are deleted.

5. The method according to claim 4, It is characterized in that If the target media timestamp is greater than the header media timestamp, obtaining a delayed media timestamp corresponding to the target media timestamp includes: If the target media timestamp is greater than the header media timestamp, obtaining a buffer delay interval; The buffering delay interval is subtracted from the target media timestamp to obtain a delayed media timestamp corresponding to the target media timestamp.

6. The method according to claim 4, It is characterized in that If the delayed media timestamp is greater than the header media timestamp, deleting the audio frames in the audio frame buffer queue whose media timestamps are greater than the delayed media timestamp, comprises: If the delayed media timestamp is greater than the header media timestamp, obtaining a delay interval between the delayed media timestamp and the header media timestamp; The number of frames to be deleted is obtained based on the delay interval, and audio frames in the audio frame buffer queue that meet the number of frames to be deleted are deleted in chronological order.

7. The method according to claim 1, It is characterized in that The method further comprises: According to the order of the audio frames in the audio frame buffer queue, the audio frames in the audio frame buffer queue are played in sequence.

8. A data synchronization device, It is characterized in that The device comprises: An audio frame receiving module, used for acquiring the audio frame sent by the sending end, and inserting the audio frame into the audio frame buffer queue; A timestamp conversion module, used for querying the current timestamp of the reference clock and converting the current timestamp into a target media timestamp; A header timestamp acquisition module, used to acquire a header media timestamp of the audio frame that is first sorted in the audio frame buffer queue; The audio frame deletion module is used to delete the audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp if the target media timestamp is greater than the header media timestamp.

9. A computer storage medium, It is characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 7.

10. An electronic device, It is characterized in that include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 7.

Images