A data synchronization method, device, storage medium and electronic device

Abstract

The application discloses a data synchronization method and device, a storage medium and an electronic device. The method comprises the following steps: acquiring an audio frame sent by a sending end, inserting the audio frame into an audio frame buffer queue, inquiring a current timestamp of a reference clock, converting the current timestamp into a target media timestamp, acquiring a head media timestamp of an audio frame with the highest priority in the audio frame buffer queue, and deleting an audio frame with a media timestamp smaller than the target media timestamp in the audio frame buffer queue if the target media timestamp is greater than the head media timestamp. According to the application, the audio frame with a relatively large delay time in the audio frame buffer queue is deleted, so that the same audio can be output at the same time point as the sending end, and the data synchronization between the sending end and the receiving end is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data synchronization method and device, storage medium and electronic equipment. BACKGROUND

[0002] AES67 is an open network digital audio standard, which is based on IP network architecture, uses existing IT network protocol, and realizes high-performance interoperability guidelines for professional audio transmission. In the prior art, AES67 protocol is often used for audio transmission between devices, so as to realize audio data synchronization between different devices, that is, different devices synchronously play the same audio data. However, due to problems such as device hardware, data transmission environment, etc., the audio transmission, audio rendering or audio playing process may be out of synchronization, resulting in different audio output by different devices at the same time, that is, different audio is output at the same time. Therefore, a data synchronization method for ensuring the synchronization of audio output by different devices is needed. SUMMARY

[0003] The embodiments of the present application provide a data synchronization method and device, storage medium and electronic equipment, which can delete audio frames with large delay time in the audio frame buffer queue, so that the same audio can be output at the same time point as the sending end, and the data synchronization between the sending end and the receiving end is improved. The technical solution is as follows:

[0004] In a first aspect, the embodiments of the present application provide a data synchronization method, which comprises the following steps:

[0005] An audio frame sent by a sending end is acquired, and the audio frame is inserted into an audio frame buffer queue;

[0006] A current timestamp of a reference clock is inquired, and the current timestamp is converted into a target media timestamp;

[0007] A head media timestamp of an audio frame with the highest priority in the audio frame buffer queue is acquired;

[0008] If the target media timestamp is greater than the head media timestamp, the audio frame with a media timestamp greater than the first target media timestamp in the audio frame buffer queue is deleted.

[0009] In a second aspect, the embodiments of the present application provide a data synchronization device, which comprises the following modules:

[0010] An audio frame receiving module is configured to acquire an audio frame sent by a sending end, and insert the audio frame into an audio frame buffer queue;

[0011] A timestamp conversion module is configured to inquire a current timestamp of a reference clock, and convert the current timestamp into a target media timestamp;

[0012] a head timestamp obtaining module, configured to obtain a head media timestamp of an audio frame at a head of the audio frame buffer queue;

[0013] an audio frame deleting module, configured to, if the target media timestamp is greater than the head media timestamp, delete an audio frame in the audio frame buffer queue with a media timestamp greater than the first target media timestamp.

[0014] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and performing the method steps described above.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, which can include a processor and a memory, and the memory stores a computer program, and the computer program is suitable for being loaded by the processor and performing the method steps described above.

[0016] In one or more embodiments of the present application, an audio frame sent by a sending end is obtained, the audio frame is inserted into an audio frame buffer queue, a current timestamp of a reference clock is inquired, the current timestamp is converted into a target media timestamp, a head media timestamp of an audio frame at a head of the audio frame buffer queue is obtained, and if the target media timestamp is greater than the head media timestamp, an audio frame in the audio frame buffer queue with a media timestamp greater than the first target media timestamp is deleted. By deleting an audio frame with a greater delay time in the audio frame buffer queue, the same audio can be output at the same time point as the sending end, and the data synchronization with the sending end is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0018] Figure 1 is an example schematic diagram of data synchronization provided by an embodiment of the present application;

[0019] Figure 2 is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0020] Figure 3 is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0021] Figure 4is a blank frame insertion example schematic diagram provided by an embodiment of the present application;

[0022] Figure 5 is a structure schematic diagram of a data synchronization device provided by an embodiment of the present application;

[0023] Figure 6 is a structure schematic diagram of a data synchronization device provided by an embodiment of the present application;

[0024] Figure 7 is a structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0026] The sending end can interact with the receiving end based on the AES67 protocol, and the sending end can send audio data to the receiving end frame by frame. After receiving the audio frame, the receiving end can insert the audio frame into an audio frame buffer queue, and play the audio frame in sequence according to the order in the audio frame buffer queue, so as to achieve the purpose of audio data synchronization between the sending end and the receiving end. It can be understood that the sending end and the receiving end can transmit audio data based on the AES67 protocol. The sending end can be connected with multiple receiving ends, and the receiving end can be connected with multiple sending ends. The sending end can send and receive audio data. Similarly, the receiving end can receive and send audio data. The sending end and the receiving end can be terminal devices such as mobile phones, computers, tablet computers and wearable devices. The data synchronization device can be the receiving end, or a module in the receiving end for implementing the data synchronization method.

[0027] Please see Figure 1An example schematic diagram of a data synchronization method is provided for the embodiments of the present application. The data synchronization device can receive the audio frame sent by the sending end. The sending end can send the audio frame 4 to the receiving end. After the data synchronization device obtains the audio frame 4 sent by the sending end, the data synchronization device can insert the audio frame 4 into the tail of the audio frame buffer queue. It can be understood that the other audio frames in the audio frame buffer queue are audio frames received before the audio frame 4 and not yet played, such as the audio frame 1, the audio frame 2 and the audio frame 3. The audio frames are received by the receiving end before the audio frame 4 is received. After the audio frames are received, the data synchronization device can also delete the audio frames with a delay time that is too long in the audio frame buffer queue. For example, after the data synchronization device receives the audio frame 4, the data synchronization device can obtain the current timestamp of the reference clock and convert the current timestamp into a corresponding target media timestamp. If the media timestamp of the audio frame 1 is less than the target media timestamp, it indicates that the sending end has played the audio frame 1, while the receiving end has not played the audio frame 1. That is, the sending end and the receiving end can be out of synchronization. Therefore, the data synchronization device can delete the audio frame 1 and remove the audio frame 1 from the audio frame buffer queue. The receiving end can continue to play from the audio frame 2, so that the receiving end and the sending end are synchronized.

[0028] The reference clock can be a standard clock, that is, a local clock of the terminal device. The local clocks of all the sending ends and all the receiving ends are the same. The timestamp of the reference clock changes with the natural growth of time. The media timestamp is a timestamp of audio data, which is used to indicate the position of the audio data played by the audio. The media timestamp of the audio frame is based on the media timestamp of the previous audio frame, the sending frequency of the sending end for sending the audio data and the sending time interval between adjacent audio frames. For example, if the sending frequency of the sending end for sending the audio data is 50 frames / s, that is, 50 audio frames are sent per second, the sending time interval between adjacent audio frames is 20 ms. If the media timestamp of the audio frame 1 is 10:00:00, the media timestamp of the audio frame 2 is 10:00:20, the media timestamp of the audio frame 3 is 10:00:40 and the media timestamp of the audio frame 4 is 10:00:60. The sending frequency can be an initial setting of the sending end or can be set by a user or a relevant staff.

[0029] The data synchronization method provided by the present application will be described in detail below in combination with specific embodiments.

[0030] Please refer to Figure 2 An example schematic diagram of a data synchronization method is provided for the embodiments of the present application. As shown in Figure 2 The method of the embodiments of the present application can include the following steps S101-S104.

[0031] S101, acquire the audio frame sent by the sending end, and insert the audio frame into an audio frame buffer queue.

[0032] Specifically, the sending end can send audio data to the receiving end according to a sending frequency, that is, the sending end sends audio frames frame by frame according to the sending frequency, the data synchronization device acquires the audio frame sent by the sending end, and then inserts the audio frame into the tail of the audio frame buffer queue. The receiving end can play the audio frames in the audio frame buffer queue in order according to the arrangement order of the audio frames in the audio frame buffer queue.

[0033] It can be understood that the data synchronization device can perform a check on the audio frame after receiving the audio frame, and if the audio frame is correct, the audio frame can be inserted into the audio frame buffer queue. The data synchronization device synchronously renders the audio frame after inserting the audio frame into the audio frame buffer queue, which facilitates the subsequent playing of the audio frame by the receiving end.

[0034] S102, query a current timestamp of a reference clock, and convert the current timestamp into a target media timestamp.

[0035] Specifically, the data synchronization device can query a current timestamp of a reference clock, that is, the data synchronization device can query the current timestamp corresponding to the time point at which the reference clock receives the audio frame, and then convert the current timestamp into a media timestamp to acquire the target media timestamp corresponding to the current timestamp. It can be understood that, since the reference clocks of the receiving end and the sending end are the same and the current timestamps are the same, the media timestamp corresponding to the current timestamp is the media timestamp of the audio played by the sending end at the current timestamp, which indicates the playing progress of the audio of the sending end.

[0036] S103, acquire the head media timestamp of the audio frame with the highest order in the audio frame buffer queue.

[0037] Specifically, the data synchronization device can acquire the media timestamp of the audio frame with the highest order in the audio frame buffer queue, and confirm it as the head media timestamp. The head media timestamp is the earliest media timestamp in the audio frame buffer queue, and the audio frame with the highest order in the audio frame buffer queue is the audio frame that will be played next by the receiving end.

[0038] S104, if the target media timestamp is greater than the head media timestamp, delete the audio frame with a media timestamp less than the target media timestamp in the audio frame buffer queue.

[0039] Specifically, if the target media timestamp is greater than the header media timestamp, it indicates an audio asynchrony between the receiving and sending ends. This means the audio playback progress at the receiving end lags behind that at the sending end, and there is at least one audio frame in the audio frame buffer queue that the sending end has already played but the receiving end has not yet. It's understandable that all audio frames in the audio frame buffer queue with media timestamps less than the target media timestamp are those that the sending end has already played but the receiving end has not. If these audio frames continue to remain in the audio frame buffer queue waiting for the receiving end to play them, it will cause a significant audio data asynchrony between the receiving and sending ends. Therefore, the data synchronization device can delete audio frames with media timestamps less than the target media timestamp from the audio frame buffer queue, thereby achieving audio data synchronization between the receiving and sending ends.

[0040] In this embodiment, an audio frame sent by the transmitting end is acquired, inserted into an audio frame buffer queue, the current timestamp of a reference clock is queried, and converted into a target media timestamp. The header media timestamp of the first audio frame in the audio frame buffer queue is obtained. If the target media timestamp is greater than the header media timestamp, audio frames in the audio frame buffer queue with media timestamps greater than the first target media timestamp are deleted. By deleting audio frames with large delays in the audio frame buffer queue, the same audio can be output at the same time as the transmitting end, improving data synchronization with the transmitting end.

[0041] Please see Figure 3 The diagram below illustrates a data synchronization method provided in this application. Figure 3 As shown, the method described in this application embodiment may include the following steps S201-S208.

[0042] S201, obtain the audio frame sent by the sending end, and obtain the received media timestamp of the audio frame.

[0043] Specifically, the sending end can send audio data to the receiving end according to a transmission frequency. That is, the sending end sends audio frames one by one according to the transmission frequency. The data synchronization device receives the audio frames sent by the sending end. It can be understood that after receiving the audio frames, the data synchronization device can perform verification processing on the audio frames. If the audio frame verification is correct, it can be inserted into the audio frame buffer queue. Simultaneously, the data synchronization device performs rendering processing on the audio frame after inserting it into the audio frame buffer queue, facilitating subsequent playback processing by the receiving end. Similarly, if the audio frame verification fails, the audio frame can be discarded, and a blank frame can be inserted into the audio frame buffer queue to fill the position of the audio frame during audio playback, thereby ensuring the integrity of the audio frames and preventing audio corruption during playback.

[0044] S202, if the audio frame buffer queue is not empty, then obtain the tail media timestamp of the last sorted audio frame in the audio frame buffer queue.

[0045] Specifically, the data synchronization device can determine whether the audio frame buffer queue is empty. If the audio frame buffer queue is not empty, that is, there are still audio frames waiting to be played in the audio frame buffer queue, the data synchronization device can obtain the media timestamp corresponding to the last audio frame in the audio frame buffer queue, that is, the tail media timestamp. The audio frame corresponding to the tail media timestamp is the previous audio frame received by the receiving end before the current audio frame.

[0046] Optionally, if the audio frame buffer queue is not empty, that is, there are no audio frames waiting to be played in the audio frame buffer queue, the data synchronization device can directly insert the audio frame into the audio frame buffer queue. It can be understood that after the audio frame is inserted into the audio frame buffer queue, the head media timestamp and the tail media timestamp of the audio frame buffer queue will both be the received media timestamp of the audio frame.

[0047] S203, if the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the transmission interval, then blank frame insertion processing is performed on the audio frame buffer queue.

[0048] Specifically, during data transmission between the sending and receiving ends, audio frame errors and loss may occur. Therefore, it is necessary to obtain the timestamp interval between the tail media timestamp and the received media timestamp. Based on the timestamp interval and the sending interval of the sending end, it is determined whether there are any dropped frames between the tail media timestamp and the received media timestamp. The sending interval is the time interval between two adjacent audio frames sent by the sending end. If the timestamp interval is greater than or equal to twice the sending interval, it indicates that there are dropped frames between the tail media timestamp and the received media timestamp. If there are dropped frames, blank frames need to be inserted into the audio frame buffer queue to make up for the audio frames lost between the tail media timestamp and the received media timestamp.

[0049] Optionally, the data synchronization device can obtain the timestamp interval between the tail media timestamp and the received media timestamp, and obtain the transmission frequency of the sending end. Based on the transmission frequency, it can calculate the transmission interval of the sending end. Based on the timestamp interval and the transmission interval, it can calculate the number of blank frames to be inserted into the audio frame buffer queue, and then insert blank frames that meet the number of blank frames into the tail of the audio frame buffer queue.

[0050] S204, insert the audio frame into the audio frame buffer queue.

[0051] Specifically, the data synchronization device can insert audio frames into the tail of the audio frame buffer queue, and the receiving end can play the audio frames in the audio frame buffer queue in sequence according to the order of the audio frames in the audio frame buffer queue.

[0052] Please see also Figure 4 This document provides an example of blank frame insertion in an embodiment of this application. If the data synchronization device receives audio frame D from the receiving end, and the received media timestamp corresponding to audio frame D is 10:00:50, and the audio frame buffer queue is not empty, containing audio frames A, B, and C waiting to be played, with corresponding media timestamps of 10:00:00, 10:00:10, and 10:00:20 respectively, then the last audio frame in the audio frame buffer queue is audio frame C, and its tail media timestamp is 10:00:20. The data synchronization device can then obtain the sent... If the transmission frequency of the sending end is 100 frames / s, that is, the sending end sends 100 audio frames per second, then the transmission interval should be 10ms. The time interval between the tail media timestamp and the received media timestamp, that is, the timestamp interval between audio frame C and audio frame D, is 30ms. Therefore, two audio frames are missing before audio frame D. The number of blank frames to be inserted is 2. Then the data synchronization device can insert the two blank frames into the tail of the audio frame buffer queue, that is, after audio frame C. After completing the blank frame insertion process, audio frame D is then inserted into the tail of the audio frame buffer queue, that is, after the blank frames.

[0053] Understandably, when an audio frame is inserted into the audio frame buffer queue, the tail media timestamp of the audio frame buffer queue changes, becoming the current media timestamp corresponding to the audio frame, such as... Figure 4 As shown, if audio frame D is inserted into the audio frame buffer queue, the tail media timestamp of the audio frame buffer queue changes from 10:00:20 to 10:00:50.

[0054] S205, query the current timestamp of the reference clock and obtain the target media timestamp corresponding to the current timestamp.

[0055] Specifically, the data synchronization device can query the current timestamp of the reference clock. That is, the data synchronization device can query the current timestamp corresponding to the time the audio frame is received, and then convert the current timestamp into a media timestamp to obtain the target media timestamp corresponding to the current timestamp. It can be understood that since the reference clocks and current timestamps of the receiving and transmitting ends are the same, the media timestamp corresponding to the current timestamp is the media timestamp of the audio played by the transmitting end at the current timestamp, indicating the playback progress of the audio at the transmitting end.

[0056] Optionally, the data synchronization device can calculate the corresponding target media timestamp T2 based on the current timestamp T1 of the reference clock, using the following formula:

[0057] T2 = (T1 × f) MODMAXOffset

[0058] Where f is the sampling frequency of the receiving end, which is also the transmission frequency of the sending end, and MAXOffset is the maximum offset value of the media timestamp, MaxOffset=0×FFFFFFFF.

[0059] S206, obtain the header media timestamp of the first audio frame in the audio frame buffer queue.

[0060] Specifically, the data synchronization device can obtain the media timestamp of the first audio frame in the audio frame buffer queue and identify it as the head media timestamp. The head media timestamp is the earliest media timestamp in the audio frame buffer queue, and the first audio frame in the audio frame buffer queue is the next audio frame that the receiving end will play.

[0061] S207, If the target media timestamp is greater than the header media timestamp, then obtain the delayed media timestamp corresponding to the target media timestamp.

[0062] Specifically, if the target media timestamp is greater than the header media timestamp, it indicates an audio asynchrony between the receiving and sending ends. This means the audio playback progress at the receiving end lags behind that at the sending end, and there is at least one audio frame in the audio frame buffer queue that the sending end has already played but the receiving end has not yet. It's understandable that all audio frames in the audio frame buffer queue with media timestamps less than the target media timestamp are those that the sending end has already played but the receiving end has not. If these audio frames continue to remain in the audio frame buffer queue waiting for the receiving end to play them, it will cause a significant audio data asynchrony between the receiving and sending ends. Therefore, the data synchronization device can delete audio frames with media timestamps less than the target media timestamp from the audio frame buffer queue, thereby achieving audio data synchronization between the receiving and sending ends.

[0063] It is understandable that data transmission takes a certain amount of time, so there will inevitably be a delay in data transmission between the receiving end and the sending end. The data synchronization device can allow a certain buffer delay interval between the audio played by the receiving end and the sending end. The buffer delay interval is the maximum delay time interval between the audio played by the receiving end and the sending end. If the target media timestamp is greater than the head media timestamp, the delayed media timestamp corresponding to the target media timestamp can be obtained. The delayed media timestamp is the media timestamp corresponding to the maximum delay interval that the data synchronization device can allow based on the target media timestamp.

[0064] Optionally, if the target media timestamp is greater than the header media timestamp, the data synchronization device can obtain a buffer delay interval. This buffer delay interval can be an initial setting of the data synchronization device or can be configured by the user or relevant personnel. For example, it can be 4ms, meaning the maximum delay between the receiver and transmitter is 4ms. The data synchronization device can subtract the buffer delay interval from the target media timestamp to obtain the delayed media timestamp corresponding to the target media timestamp. For example, if the target media timestamp is 10:00:00 and the buffer delay interval is 4ms, then the corresponding buffer delayed timestamp is 10:00:04.

[0065] S208, if the delayed media timestamp is greater than the head media timestamp, then delete audio frames in the audio frame buffer queue whose media timestamp is less than the delayed media timestamp.

[0066] Specifically, if the delayed media timestamp is still greater than the head media timestamp, the data synchronization device can delete audio frames in the audio frame buffer queue whose media timestamps are greater than the delayed media timestamps. It is understood that if the delayed media timestamp is less than or equal to the head media timestamp, the data synchronization device does not need to delete audio frames in the audio frame buffer queue.

[0067] Optionally, if the delayed media timestamp is greater than the header media timestamp, the delay interval between the delayed media timestamp and the header media timestamp is obtained. The delay interval is the difference between the delayed media timestamp and the header media timestamp. The data synchronization device can obtain the number of frames to be deleted based on the delay interval. In other words, the data synchronization device can calculate the number of frames to be deleted based on the delay interval and the sending frequency of the sending end. The number of frames to be deleted is the number of audio frames in the audio frame buffer queue whose media timestamp is less than the delayed media timestamp. The data synchronization device can delete audio frames in the audio frame buffer queue that meet the data to be deleted in chronological order. The data synchronization device can sort the audio frames in the audio frame buffer queue according to the number of audio frames to be deleted before deletion.

[0068] Optionally, the data synchronization device can control the receiving end to play the audio frames in the audio frame buffer queue sequentially according to the order of the audio frames in the audio frame buffer queue.

[0069] In this embodiment, the audio frame sent by the transmitting end is obtained, and the received media timestamp of the audio frame is obtained. The data synchronization device can perform verification processing on the audio frame to ensure that the audio frame is intact and avoid audio corruption during audio playback. If the audio frame buffer queue is not empty, the tail media timestamp of the last audio frame in the audio frame buffer queue is obtained. Based on the timestamp interval between the tail media timestamp and the received media timestamp, blank frame insertion processing is performed on the audio frame buffer queue. If there are dropped frames, blank frame insertion processing is performed on the audio frame buffer queue to make up for the audio frames lost between the tail media timestamp and the received media timestamp. The audio frame is inserted into the audio frame buffer queue. The current timestamp of the reference clock is queried and converted into the target media timestamp. If the target media timestamp is greater than the head media timestamp, the delayed media timestamp corresponding to the target media timestamp is obtained. If the delayed media timestamp is still greater than the head media timestamp, the audio frames in the audio frame buffer queue with media timestamps less than delayed media timestamps are deleted. The data synchronization device can control the receiving end to play audio frames sequentially according to their order in the audio frame buffer queue. By deleting audio frames with large delays from the audio frame buffer queue, it ensures that the receiving end outputs the same audio at the same time, thus improving data synchronization with the sending end.

[0070] The following will be combined with the appendix Figure 5 - Appendix Figure 6 This application provides a detailed description of the data synchronization device provided in its embodiments. It should be noted that the appendix... Figure 5 - Appendix Figure 6 The data synchronization device in the present application is used to perform the data synchronization function. Figure 2 and Figure 3 The methods shown in the embodiments are for illustrative purposes only, illustrating the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this application. Figure 2 and Figure 3 The example shown.

[0071] Please see Figure 5 This illustration shows a schematic diagram of a data synchronization device provided in an exemplary embodiment of this application. The data synchronization device can be implemented as all or part of a device through software, hardware, or a combination of both. The device 1 includes an audio frame receiving module 11, a timestamp conversion module 12, a header timestamp acquisition module 13, and an audio frame deletion module 14.

[0072] The audio frame receiving module 11 is used to acquire the audio frame sent by the sending end and insert the audio frame into the audio frame buffer queue.

[0073] The timestamp conversion module 12 is used to query the current timestamp of the reference clock and convert the current timestamp into the target media timestamp;

[0074] Head timestamp acquisition module 13 is used to acquire the head media timestamp of the first audio frame in the audio frame buffer queue.

[0075] The audio frame deletion module 14 is used to delete audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp if the target media timestamp is greater than the header media timestamp.

[0076] In this embodiment, an audio frame sent by the transmitting end is acquired, inserted into an audio frame buffer queue, the current timestamp of the reference clock is queried, and converted into a target media timestamp. The header media timestamp of the first audio frame in the audio frame buffer queue is obtained. If the target media timestamp is greater than the header media timestamp, audio frames in the audio frame buffer queue with media timestamps greater than the first target media timestamp are deleted. By deleting audio frames with large delays in the audio frame buffer queue, the same audio can be output at the same time as the transmitting end, improving data synchronization with the transmitting end.

[0077] Please see Figure 6This illustration shows a schematic diagram of a data synchronization device provided in an exemplary embodiment of this application. The data synchronization device can be implemented as all or part of a device through software, hardware, or a combination of both. It includes an audio frame receiving module 11, a timestamp conversion module 12, a header timestamp acquisition module 13, an audio frame deletion module 14, and an audio frame playback module 15.

[0078] The audio frame receiving module 11 is used to acquire the audio frame sent by the sending end and insert the audio frame into the audio frame buffer queue.

[0079] Optionally, the audio frame receiving module 11 is specifically used to obtain the audio frame sent by the sending end and obtain the received media timestamp of the audio frame;

[0080] If the audio frame buffer queue is not empty, then obtain the tail media timestamp of the last sorted audio frame in the audio frame buffer queue;

[0081] If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the transmission interval, then blank frame insertion processing is performed on the audio frame buffer queue.

[0082] Insert the audio frame into the audio frame buffer queue.

[0083] Optionally, the audio frame receiving module 11 is specifically used to obtain the timestamp interval between the tail media timestamp and the received media timestamp, and to obtain the transmission frequency of the sending end;

[0084] Based on the transmission frequency, calculate the transmission interval of the transmitting end;

[0085] If the timestamp interval is greater than or equal to twice the transmission interval, then the number of inserted frames is obtained based on the timestamp interval and the transmission interval;

[0086] Insert blank frames that meet the specified number of insertion frames into the tail of the audio frame buffer queue.

[0087] The timestamp conversion module 12 is used to query the current timestamp of the reference clock and convert the current timestamp into the target media timestamp;

[0088] Head timestamp acquisition module 13 is used to acquire the head media timestamp of the first audio frame in the audio frame buffer queue.

[0089] The audio frame deletion module 14 is used to delete audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp if the target media timestamp is greater than the header media timestamp.

[0090] Optionally, the audio frame deletion module 14 is specifically used to obtain the delayed media timestamp corresponding to the target media timestamp if the target media timestamp is greater than the header media timestamp;

[0091] If the delayed media timestamp is greater than the header media timestamp, then audio frames in the audio frame buffer queue with media timestamps greater than the delayed media timestamp are deleted.

[0092] Optionally, the audio frame deletion module 14 is specifically used to obtain the buffer delay interval if the target media timestamp is greater than the header media timestamp;

[0093] The buffer delay interval is subtracted from the target media timestamp to obtain the delayed media timestamp corresponding to the target media timestamp.

[0094] Optionally, the audio frame deletion module 14 is specifically used to obtain the delay interval between the delayed media timestamp and the head media timestamp if the delayed media timestamp is greater than the head media timestamp;

[0095] The number of frames to be deleted is obtained based on the delay interval, and the audio frames in the audio frame buffer queue that meet the number of frames to be deleted are deleted in chronological order.

[0096] The audio frame playback module 15 is used to play the audio frames in the audio frame buffer queue sequentially according to the order of the audio frames in the audio frame buffer queue.

[0097] In this embodiment, the audio frames sent by the transmitting end are obtained, and the received media timestamp of the audio frames is also obtained. The data synchronization device can perform verification processing on the audio frames to ensure their integrity and prevent audio corruption during playback. If the audio frame buffer queue is not empty, the tail media timestamp of the last audio frame in the queue is obtained. Based on the timestamp interval between the tail media timestamp and the received media timestamp, blank frames are inserted into the audio frame buffer queue. If there are dropped frames, blank frames are inserted into the audio frame buffer queue to fill the missing audio frames between the tail media timestamp and the received media timestamp. The audio frames are then inserted into the audio frame buffer queue. The current timestamp of the reference clock is queried and converted into a target media timestamp. If the target media timestamp is greater than the head media timestamp, the delayed media timestamp corresponding to the target media timestamp is obtained. If the delayed media timestamp is still greater than the head media timestamp, audio frames in the audio frame buffer queue with media timestamps less than the delayed media timestamp are deleted. The data synchronization device can control the receiving end to play audio frames sequentially according to their order in the audio frame buffer queue. By deleting audio frames with large delays from the audio frame buffer queue, it ensures that the receiving end outputs the same audio at the same time, thus improving data synchronization with the sending end.

[0098] It should be noted that the data synchronization device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the data synchronization method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data synchronization device and the data synchronization method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0099] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] This application also provides a computer storage medium that can store multiple instructions, which are adapted to be loaded and executed by a processor as described above. Figures 1-4 The data synchronization method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0101] This application also provides a computer program product storing at least one instruction, which is loaded and executed by the processor as described above. Figures 1-4 The data synchronization method described in the illustrated embodiment can be found in the following documentation for its specific execution process.Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0102] Please refer to Figure 7 This diagram illustrates a structural block diagram of an electronic device provided in an exemplary embodiment of this application. The electronic device in this application may include one or more components such as a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected via the bus 150.

[0103] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device using various interfaces and lines, and executes various functions of terminal 100 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user page, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0104] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described above, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems.

[0105] The memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, while native and third-party applications run in user space. To ensure that different third-party applications can achieve good running performance, the operating system allocates corresponding system resources for each application. However, different application scenarios within the same third-party application have different requirements for system resources. For example, in local resource loading scenarios, third-party applications have high requirements for disk read speed; in animation rendering scenarios, third-party applications have high requirements for GPU performance. Since the operating system and third-party applications are independent of each other, the operating system often cannot promptly perceive the current application scenario of a third-party application, resulting in the operating system's inability to adapt system resources accordingly.

[0106] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0107] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 can be a touch display screen.

[0108] The touch display screen can be designed as a full-screen, curved screen, or irregularly shaped screen. It can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; however, this application does not limit the specific design in this regard.

[0109] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, Wireless Fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0110] exist Figure 7 In the illustrated electronic device, the processor 110 can be used to call the data synchronization application stored in the memory 120 and specifically perform the following operations:

[0111] Obtain the audio frame sent by the sending end and insert the audio frame into the audio frame buffer queue;

[0112] Query the current timestamp of the reference clock and convert the current timestamp to the target media timestamp;

[0113] Obtain the header media timestamp of the first audio frame in the audio frame buffer queue;

[0114] If the target media timestamp is greater than the header media timestamp, then audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamp are deleted.

[0115] In one embodiment, when the processor 110 performs the operation of acquiring an audio frame sent by the sender and inserting the audio frame into the audio frame buffer queue, it specifically performs the following operations:

[0116] Obtain the audio frame sent by the sending end, and obtain the received media timestamp of the audio frame;

[0117] If the audio frame buffer queue is not empty, then obtain the tail media timestamp of the last sorted audio frame in the audio frame buffer queue;

[0118] If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the transmission interval, then blank frame insertion processing is performed on the audio frame buffer queue.

[0119] Insert the audio frame into the audio frame buffer queue.

[0120] In one embodiment, when the processor 110 performs blank frame insertion processing on the audio frame buffer based on the timestamp interval between the tail media timestamp and the received media timestamp, it specifically performs the following operations:

[0121] Obtain the timestamp interval between the tail media timestamp and the received media timestamp, and obtain the transmission frequency of the sending end;

[0122] Based on the transmission frequency, calculate the transmission interval of the transmitting end;

[0123] If the timestamp interval is greater than or equal to twice the transmission interval, then the number of inserted frames is obtained based on the timestamp interval and the transmission interval;

[0124] Insert blank frames that meet the specified number of insertion frames into the tail of the audio frame buffer queue.

[0125] In one embodiment, when the processor 110 performs the operation of deleting audio frames in the audio frame buffer queue whose media timestamps are less than the target media timestamp if the target media timestamp is greater than the header media timestamp, it specifically performs the following operations:

[0126] If the target media timestamp is greater than the header media timestamp, then obtain the delayed media timestamp corresponding to the target media timestamp;

[0127] If the delayed media timestamp is greater than the header media timestamp, then audio frames in the audio frame buffer queue with media timestamps greater than the delayed media timestamp are deleted.

[0128] In one embodiment, when the processor 110 executes the operation of obtaining the delayed media timestamp corresponding to the target media timestamp if the target media timestamp is greater than the header media timestamp, it specifically performs the following operations:

[0129] If the target media timestamp is greater than the header media timestamp, then obtain the buffer delay interval;

[0130] The buffer delay interval is subtracted from the target media timestamp to obtain the delayed media timestamp corresponding to the target media timestamp.

[0131] In one embodiment, when the processor 110 performs the operation of deleting audio frames in the audio frame buffer queue whose media timestamps are greater than the delayed media timestamp if the delayed media timestamp is greater than the header media timestamp, it specifically performs the following operations:

[0132] If the delayed media timestamp is greater than the header media timestamp, then obtain the delay interval between the delayed media timestamp and the header media timestamp;

[0133] The number of frames to be deleted is obtained based on the delay interval, and the audio frames in the audio frame buffer queue that meet the number of frames to be deleted are deleted in chronological order.

[0134] In one embodiment, when executing the data synchronization method, the processor 110 also performs the following operations:

[0135] The audio frames in the audio frame buffer queue are sorted and played sequentially.

[0136] In this embodiment, the audio frames sent by the transmitting end are obtained, and the received media timestamp of the audio frames is also obtained. The data synchronization device can perform verification processing on the audio frames to ensure their integrity and prevent audio corruption during playback. If the audio frame buffer queue is not empty, the tail media timestamp of the last audio frame in the queue is obtained. Based on the timestamp interval between the tail media timestamp and the received media timestamp, blank frames are inserted into the audio frame buffer queue. If there are dropped frames, blank frames are inserted into the audio frame buffer queue to fill the missing audio frames between the tail media timestamp and the received media timestamp. The audio frames are then inserted into the audio frame buffer queue. The current timestamp of the reference clock is queried and converted into a target media timestamp. If the target media timestamp is greater than the head media timestamp, the delayed media timestamp corresponding to the target media timestamp is obtained. If the delayed media timestamp is still greater than the head media timestamp, audio frames in the audio frame buffer queue with media timestamps less than the delayed media timestamp are deleted. The data synchronization device can control the receiving end to play audio frames sequentially according to their order in the audio frame buffer queue. By deleting audio frames with large delays from the audio frame buffer queue, it ensures that the receiving end outputs the same audio at the same time, thus improving data synchronization with the sending end.

[0137] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0138] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

[0139] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the audio frames and the current timestamp of the reference clock involved in this specification were obtained with full authorization.

Claims

1. A data synchronization method, characterized in that, The method includes: Obtain the audio frame sent by the sending end and insert the audio frame into the audio frame buffer queue; Query the current timestamp of the reference clock and convert the current timestamp to the target media timestamp; Obtain the header media timestamp of the first audio frame in the audio frame buffer queue; If the target media timestamp is greater than the header media timestamp, then audio frames in the audio frame buffer queue with media timestamps less than the target media timestamp are deleted to synchronize the audio data between the receiving end and the sending end. The media timestamp of the audio frame is used to indicate the position of the audio data during playback. It is obtained by adding the transmission time interval between adjacent audio frames to the media timestamp of the previous audio frame, based on the transmission frequency of the transmitting end.

2. The method according to claim 1, characterized in that, The step of obtaining the audio frame sent by the sending end and inserting the audio frame into the audio frame buffer queue includes: Obtain the audio frame sent by the sending end, and obtain the received media timestamp of the audio frame; If the audio frame buffer queue is not empty, then obtain the tail media timestamp of the last sorted audio frame in the audio frame buffer queue; If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the transmission interval, then blank frame insertion processing is performed on the audio frame buffer queue. Insert the audio frame into the audio frame buffer queue.

3. The method according to claim 2, characterized in that, If the timestamp interval between the tail media timestamp and the received media timestamp is greater than or equal to twice the transmission interval, then blank frame insertion processing is performed on the audio frame buffer queue, including: Obtain the timestamp interval between the tail media timestamp and the received media timestamp, and obtain the transmission frequency of the sending end; Based on the transmission frequency, calculate the transmission interval of the transmitting end; If the timestamp interval is greater than or equal to twice the transmission interval, then the number of inserted frames is obtained based on the timestamp interval and the transmission interval; Insert blank frames that meet the specified number of insertion frames into the tail of the audio frame buffer queue.

4. The method according to claim 1, characterized in that, If the target media timestamp is greater than the header media timestamp, then the audio frames in the audio frame buffer queue with media timestamps less than the target media timestamp are deleted, including: If the target media timestamp is greater than the header media timestamp, then obtain the delayed media timestamp corresponding to the target media timestamp; If the delayed media timestamp is greater than the header media timestamp, then audio frames in the audio frame buffer queue with media timestamps greater than the delayed media timestamp are deleted.

5. The method according to claim 4, characterized in that, If the target media timestamp is greater than the header media timestamp, then obtaining the delayed media timestamp corresponding to the target media timestamp includes: If the target media timestamp is greater than the header media timestamp, then obtain the buffer delay interval; The buffer delay interval is subtracted from the target media timestamp to obtain the delayed media timestamp corresponding to the target media timestamp.

6. The method according to claim 4, characterized in that, If the delayed media timestamp is greater than the header media timestamp, then the audio frames in the audio frame buffer queue with media timestamps greater than the delayed media timestamp are deleted, including: If the delayed media timestamp is greater than the header media timestamp, then obtain the delay interval between the delayed media timestamp and the header media timestamp; The number of frames to be deleted is obtained based on the delay interval, and the audio frames in the audio frame buffer queue that meet the number of frames to be deleted are deleted in chronological order.

7. The method according to claim 1, characterized in that, The method further includes: The audio frames in the audio frame buffer queue are sorted and played sequentially.

8. A data synchronization device, characterized in that, The device includes: An audio frame receiving module is used to acquire audio frames sent by the sending end and insert the audio frames into an audio frame buffer queue. The timestamp conversion module is used to query the current timestamp of the reference clock and convert the current timestamp into the target media timestamp; The header timestamp acquisition module is used to acquire the header media timestamp of the first audio frame in the audio frame buffer queue. An audio frame deletion module is used to delete audio frames in the audio frame buffer queue whose media timestamps are greater than the first target media timestamp if the target media timestamp is greater than the header media timestamp, so as to synchronize the audio data between the receiving end and the sending end. The media timestamp of the audio frame is used to indicate the position of the audio data played, and is obtained by adding the transmission time interval between adjacent audio frames based on the media timestamp of the previous audio frame, the transmission frequency of the audio data sent by the sending end, and the transmission time interval between adjacent audio frames.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are adapted to be loaded by a processor and executed as method steps as claimed in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 7.

Images