Data synchronization method, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of communication, and provides a data synchronization method, electronic equipment and a computer readable storage medium, the method comprises the following steps: obtaining a first sampling rate corresponding to first to-be-synchronized data and a second sampling rate corresponding to second to-be-synchronized data, the first to-be-synchronized data comprising multiple frames of first data, and the second to-be-synchronized data comprising multiple frames of second data; the second to-be-synchronized data comprises multiple frames of second data, and the first sampling rate is higher than the second sampling rate. According to the first sampling rate and the second sampling rate, the segmentation time is determined, and a first overflow period corresponding to the first sampling rate and a second overflow period corresponding to the second sampling rate are integer multiples of the segmentation time. And determining a target time difference according to the segmentation time, the first sampling rate, the second sampling rate, the first timestamp and the second timestamp. And performing synchronization processing on the first to-be-synchronized data and the second to-be-synchronized data according to the target time difference. The time difference between the first to-be-synchronized data and the second to-be-synchronized data can be determined, the data are synchronized, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of communication technologies, and in particular, to a data synchronization method, an electronic device, and a computer-readable storage medium. Background Art

[0002] RTP (Real-time Transport Protocol), MPEG-TS (MPEG-2 Transport Stream), FLV (Flash Video), and MP4 (MPEG-4 Part 14) can be applied to the field of transmission of real-time multimedia data such as network audio and video.

[0003] However, during the encoding and transmission processes, audio data and video data each have independent time axes. Even if the sending times are the same, factors such as network latency can still cause a deviation in the receiving time, thereby causing the problem of audio-video desynchronization.

[0004] In addition, since the timestamp of a data packet is usually 32 bits and will start counting from 0 again after reaching the maximum value (i.e., the overflow phenomenon). Since the sampling rates of audio data and video data are usually different, their timestamp overflow periods are also different, resulting in the playback device being unable to directly determine the relative time difference between audio and video through timestamps. In a system running for a long time, the error of the timestamp will gradually accumulate, further exacerbating the problem of audio-video desynchronization. Summary of the Invention

[0005] Embodiments of this application provide a data synchronization method, apparatus, chip, electronic device, and computer-readable storage medium, which can determine the time difference between audio-video data, thereby performing synchronization processing on the audio-video data to achieve lip-sync and improve the user's audio-visual experience.

[0006] In a first aspect, the present application provides a data synchronization method, including: obtaining a first sampling rate corresponding to first data to be synchronized, and obtaining a second sampling rate corresponding to second data to be synchronized, where the first data to be synchronized includes multiple frames of first data, the second data to be synchronized includes multiple frames of second data, and the first sampling rate is higher than the second sampling rate. Determining a segmentation time according to the first sampling rate and the second sampling rate, where a first overflow period corresponding to the first sampling rate is an integer multiple of the segmentation time, and a second overflow period corresponding to the second sampling rate is an integer multiple of the segmentation time. Determining a target time difference according to the segmentation time, the first sampling rate, the second sampling rate, a first timestamp and a second timestamp, where the first timestamp is a timestamp corresponding to a first data frame, the second timestamp is a timestamp corresponding to a second data frame, the first data frame is any one of the multiple frames of first data, and the second data frame is a frame of the multiple frames of second data corresponding to the first data frame. Synchronizing the first data to be synchronized and the second data to be synchronized according to the target time difference.

[0007] In some embodiments, determining a target time difference according to the segmentation time, the first sampling rate, the second sampling rate, the first timestamp and the second timestamp includes: determining a first relative time according to the first timestamp, the first sampling rate and the segmentation time. Determining a second relative time according to the second timestamp, the second sampling rate and the segmentation time. Determining a first time difference according to the first relative time and the second relative time. Determining a second time difference and a third time difference according to the first time difference and the segmentation time. Obtaining a target time difference threshold. Determining the target time difference according to the target time difference threshold, the absolute value of the first time difference, the absolute value of the second time difference and the absolute value of the third time difference.

[0008] In some embodiments, determining a first relative time according to the first timestamp, the first sampling rate and the segmentation time includes: determining a ratio of the first timestamp to the first sampling rate as a first ratio. Determining a remainder of the ratio of the first ratio to the segmentation time as the first relative time. Determining a second relative time according to the second timestamp, the second sampling rate and the segmentation time includes: determining a ratio of the second timestamp to the second sampling rate as a second ratio. Determining a remainder of the ratio of the second ratio to the segmentation time as the second relative time.

[0009] In some embodiments, determining a first time difference according to the first relative time and the second relative time includes: determining a difference between the second relative time and the first relative time as the first time difference. Determining a second time difference and a third time difference according to the first time difference and the segmentation time includes: determining a sum of the first time difference and the segmentation time as the second time difference. Determining a difference between the first time difference and the segmentation time as the third time difference.

[0010] In some embodiments, obtaining the target time difference threshold includes: obtaining a preset time threshold. Determining half of the segmentation time as the first time threshold. Determining the target time difference threshold according to the preset time threshold and the first time threshold, where the target time difference threshold is the minimum value of the preset time threshold and the first time threshold.

[0011] In some embodiments, the target time difference is the one among the absolute value of the first time difference, the absolute value of the second time difference, and the absolute value of the third time difference that is less than the target time difference threshold.

[0012] In some embodiments, determining the segmentation time according to the first sampling rate and the second sampling rate includes: determining the greatest common divisor of the first sampling rate and the second sampling rate as the sampling rate common divisor. Determining the ratio of the second sampling rate to the sampling rate common divisor as the synchronization period. Determining the ratio of the first overflow period to the synchronization period as the segmentation time.

[0013] In some embodiments, synchronizing the first data to be synchronized and the second data to be synchronized according to the target time difference includes: if the target time difference is positive, pausing the playback of the second data to be synchronized within the target time, where the target time is the time corresponding to the target time difference. If the target time difference is negative, playing the third data frame, where the third timestamp corresponding to the third data frame is the timestamp after the second timestamp and is separated from the second timestamp by the target time, and the third data frame is one of multiple frames of the second data.

[0014] In some embodiments, the first data to be synchronized is video data and the first sampling rate is the video sampling rate. The second data to be synchronized is audio data and the second sampling rate is the audio sampling rate.

[0015] In a second aspect, the present application provides a data synchronization device, including:

[0016] An acquisition module, configured to acquire the first sampling rate corresponding to the first data to be synchronized, and acquire the second sampling rate corresponding to the second data to be synchronized. The first data to be synchronized includes multiple frames of first data, and the second data to be synchronized includes multiple frames of second data, and the first sampling rate is higher than the second sampling rate.

[0017] A processing module, configured to determine the segmentation time according to the first sampling rate and the second sampling rate, where the first overflow period corresponding to the first sampling rate is an integer multiple of the segmentation time, and the second overflow period corresponding to the second sampling rate is an integer multiple of the segmentation time.

[0018] The processing module is further configured to determine a target time difference according to the segmentation time, the first sampling rate, the second sampling rate, the first timestamp, and the second timestamp. The first timestamp is the timestamp corresponding to the first data frame, the second timestamp is the timestamp corresponding to the second data frame, the first data frame is any one of multiple frames of first data, and the second data frame is the frame of data corresponding to the first data frame in multiple frames of second data.

[0019] The processing module is further configured to perform synchronization processing on the first data to be synchronized and the second data to be synchronized according to the target time difference.

[0020] In a third aspect, the present application provides a chip, which is used to execute the method in any one of the above first aspects.

[0021] In a fourth aspect, the present application provides an electronic device, including a processor and a memory. The processor is configured to execute a computer program stored in the memory to implement the method in any one of the above first aspects. Or,

[0022] The electronic device includes the chip as in the third aspect.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method in any one of the above first aspects.

[0024] In the technical solution provided by the embodiment of the present application, the playback device can obtain the first sampling rate corresponding to the first data to be synchronized, and obtain the second sampling rate corresponding to the second data to be synchronized. The first data to be synchronized includes multiple frames of first data, the second data to be synchronized includes multiple frames of second data, and the first sampling rate is higher than the second sampling rate. And according to the first sampling rate and the second sampling rate, determine the segmentation time. The first overflow period corresponding to the first sampling rate is an integer multiple of the segmentation time, and the second overflow period corresponding to the second sampling rate is an integer multiple of the segmentation time. Then, according to the segmentation time, the first sampling rate, the second sampling rate, the first timestamp, and the second timestamp, determine the target time difference. The first timestamp is the timestamp corresponding to the first data frame, the second timestamp is the timestamp corresponding to the second data frame, the first data frame is any one of multiple frames of first data, and the second data frame is the frame of data corresponding to the first data frame in multiple frames of second data. Finally, according to the target time difference, perform synchronization processing on the first data to be synchronized and the second data to be synchronized. The technical solution provided by the embodiment of the present application can accurately adjust the synchronization relationship between the first data to be synchronized and the second data to be synchronized by analyzing the target time difference, and finally realize data synchronization, significantly improving the user experience. The technical solution provided by the embodiment of the present application ensures the accuracy and reliability of the synchronization processing through reasonable sampling rate analysis and timestamp calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0026] Figure 1 is a schematic structural diagram of an RTP data packet provided by an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of an audio - video timeline provided by an embodiment of the present application;

[0028] Figure 3 is another schematic diagram of an audio - video timeline provided by an embodiment of the present application;

[0029] Figure 4 is yet another schematic diagram of an audio - video timeline provided by an embodiment of the present application;

[0030] Figure 5 is a schematic flowchart of a method for determining the audio - video time difference provided by an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of an audio - video timestamp overflow period provided by an embodiment of the present application;

[0032] Figure 7 is another schematic diagram of an audio - video timestamp overflow period provided by an embodiment of the present application;

[0033] Figure 8 is a schematic flowchart of a data synchronization method provided by an embodiment of the present application;

[0034] Figure 9 is a schematic structural diagram of a data synchronization device provided by an embodiment of the present application;

[0035] Figure 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0036] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well - known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0037] It should be understood that, as used in the specification of this application and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their combinations.

[0038] It should also be understood that the term "and / or" as used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0039] As used in the specification of this application and the appended claims, the term "if" can be interpreted, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrases "if determined" or "if [the described condition or event] is detected" can be interpreted, depending on the context, as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0040] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are used only for differentiating descriptions and should not be construed as indicating or implying relative importance.

[0041] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0042] RTP is a protocol for real-time data transmission over a network and is widely used in various real-time data transmission scenarios that require low latency and high reliability. For example, it can be applied to scenarios that require real-time transmission of audio and video data (audio data and video data) in various fields such as audio and video communication fields, streaming media fields, monitoring and security fields, remote education and training fields, game and virtual reality fields, etc. RTP can provide end-to-end real-time data transmission services for these scenarios.

[0043] The format of an RTP packet consists of two parts: the RTP header and the RTP payload. The RTP header contains information for data transmission and control, and the RTP payload is used to carry the actual audio and video data.

[0044] Among them, the RTP header can include the fields shown in Table 1:

[0045] Table 1

[0046]

[0047] As shown in Table 1, the RTP header can include multiple fields. Among them, the version V is used to represent the version number of the RTP protocol, with bit positions 0 - 1 in the RTP packet and a length of 2 bits. The padding P is used to indicate whether there are padding bytes at the end of the RTP packet. If P is 1, it means there are padding bytes at the end of the RTP packet. The padding P has a bit position of 2 in the RTP packet and a length of 1 bit. The extension X is used to indicate whether the RTP packet has an extension header. If X is 1, it means there is an extension header in the RTP packet. The X has a bit position of 3 in the RTP packet and a length of 1 bit. The CSRC count is used to represent the number of CSRCs (Contributing Source Identifiers), and the number range of CSRCs is 0 - 15. The CSRC has bit positions 4 - 7 in the RTP packet and a length of 4 bits. The marker M is used to mark important events (such as the end of a video frame), with a bit position of 8 in the RTP packet and a length of 1 bit. The payload type PT is used to indicate the format of the RTP payload (such as audio coding type, video coding type), with bit positions 9 - 15 in the RTP packet and a length of 7 bits. The sequence number Sequence Number is used to detect packet loss and packet sorting, with bit positions 16 - 31 in the RTP packet and a length of 16 bits. The timestamp Timestamp is used to represent the sampling time of the first byte in the packet for audio and video synchronization, with bit positions 32 - 63 in the RTP packet and a length of 32 bits. The synchronization source identifier SSRC is used to uniquely identify the source of the RTP data stream, with bit positions 64 - 95 in the RTP packet and a length of 32 bits. The list of contributing source identifiers is used to identify multiple sources in a mixed stream, with bit positions 96 and later in the RTP packet and a length of 0 to 15 32 - bit fields.

[0048] The RTP payload is used to carry the actual audio and video data, and its format depends on the specific coding type. The RTP payload usually includes: audio and video frame data (encoded audio and video data), fragmentation information (if the audio and video frame is large, it may be fragmented for transmission), and padding bytes (if the padding bit (P) in the RTP header is 1, the payload may contain padding bytes at the end).

[0049] As Figure 1 shown, it is a schematic diagram of the structure of an RTP data packet provided by an embodiment of the present application. As Figure 1 shown, in addition to the RTP header, the RTP data packet may further include an extension header. If the extension X is 1, the RTP data contains an extension header. The extension header includes an extension identifier (Profile) and extension data (Data). And it includes the RTP payload, which can be expressed as Payload Data and is used to carry actual audio and video data.

[0050] As Figure 1 shown, the timestamp of the RTP data packet records the sampling moment of the audio and video frames. The playback device (the receiving end of the audio and video data) can calculate the actual playback time of the audio and video frames through the timestamp.

[0051] During the process of data transmission, the traditional technical solution usually defaults that the timestamps corresponding to the first frame of audio data and the first frame of video data sent by the sending end are the same, that is, it is default that the first frame of video and the first frame of audio are in an aligned state to ensure that the first frame of video and the first frame of audio received by the receiving end should also be in an aligned state. That is, when the audio and video reception times are the same, the audio and video will be played synchronously during playback, and the time difference between the time axes of the audio and video is 0 ms.

[0052] For example, Figure 2 it is a schematic diagram of the audio and video time axes provided by an embodiment of the present application. As Figure 2 shown, when there are 25 video frames and 50 audio frames per second. Each video frame takes 40 ms, and each audio frame takes 20 ms. It can be understood that for each video frame played by the receiving end, two audio frames need to be played correspondingly, that is, one video frame corresponds to two audio frames. When the audio and video are played synchronously, the time difference between the audio and video is 0 ms.

[0053] However, when the audio generation cycles are inconsistent, or when the audio data is temporarily missing due to network delay during the sending or receiving process. There will be a certain time difference between the time axes of the audio and video, and there will be an out-of-sync problem between the audio and video received by the receiving end.

[0054] For example, Figure 3 it is another schematic diagram of the audio and video time axes provided by an embodiment of the present application. As Figure 3As shown, when the sending device (the sending end of audio and video data) sends audio and video data simultaneously, due to network latency, the audio data starts to be sent 10 ms after the video data is sent. There will be a 10-ms time difference between the audio and video data. That is, there will always be a 10-ms time difference in the audio and video data received by the playback device during playback, and the audio-video out-of-sync phenomenon will persist with a 10-ms time difference.

[0055] Furthermore, the greater the time difference between the audio and video data, the more serious the audio-video out-of-sync phenomenon, and the worse the user's audio-visual experience.

[0056] For example, Figure 4 This is another audio-video timeline schematic diagram provided by the embodiments of this application. As Figure 4 shown, when the sending device sends audio and video data simultaneously, due to network latency, the audio data starts to be sent 20 ms after the video data is sent. There will be a 20-ms time difference between the audio and video. That is, there will always be a 20-ms time difference in the audio and video data received by the playback device during playback, and the audio-video out-of-sync phenomenon will persist with a 20-ms time difference.

[0057] It can be understood that if the playback device can determine the time difference that exists in the audio and video data during playback, it can synchronize the audio data and video data according to this time difference to achieve audio-video synchronization and improve the user's audio-visual experience.

[0058] However, the timestamp of the RTP data packet is 32 bits, and its maximum recorded value is 4,294,967,296. That is, the timestamp of the RTP data packet can record at most 4,294,967,296 unit times. When the time that needs to be recorded by the timestamp exceeds 4,294,967,296 bits, there will be a timestamp overflow problem. For example, when the sampling device samples video at a video sampling rate of 90 kHz, during the storage of the sampled video data, the timestamp corresponding to each video frame occupies 1 / 90000 seconds. When the video data is stored in the form of RTP data packets, the timestamp will overflow once every about 13.256 hours. If the sampling device samples audio at an audio sampling rate of 8 kHz, during the storage of the sampled audio data, the timestamp corresponding to each audio frame occupies 1 / 8000 seconds. When the audio data is stored in the form of RTP data packets, the timestamp will overflow once every about 149.1308 hours.

[0059] It is understandable that when the audio sampling rate is 44.1 kHz, the timestamp corresponding to each audio frame occupies 1 / 44100 seconds; when the audio sampling rate is 32 kHz, the timestamp corresponding to each audio frame occupies 1 / 32000 seconds; when the audio sampling rate is 64 kHz, the timestamp corresponding to each audio frame occupies 1 / 64000 seconds. The timestamp overflow periods corresponding to different audio sampling rates are different, which will not be elaborated in this application.

[0060] Since the RTP data packet timelines of audio data and video data are independent of each other, and the sampling rates of audio and video data are different, and the corresponding overflow periods of audio and video data are different, the playback device cannot directly determine the time difference between audio and video data through the timestamps corresponding to the RTP data packets of audio and video data, and cannot perform synchronization processing on audio and video data, so the problem of audio and video out-of-sync will continue to exist.

[0061] In traditional technical solutions, the playback device can supplement synchronization information through the 64-bit timestamp in the RTCP protocol (Real-time Transport Control Protocol) to determine the time difference between audio and video data and achieve audio and video synchronization. However, since the implementation of the RTCP protocol is relatively complex, and its core functions (such as flow control and congestion control) are not necessary in most scenarios, some playback devices are not adapted to the RTCP protocol and only support the RTP protocol.

[0062] In summary, in the prior art, the RTP timestamp cannot solve the problem of audio and video data out-of-sync, and the complexity and insufficient adaptation of the RTCP protocol limit its application, and the first-frame alignment strategy has inherent defects. Therefore, how to achieve efficient and accurate audio and video synchronization without relying on the RTCP protocol has become a technical problem to be solved urgently.

[0063] In view of this, the embodiments of the present application provide a data synchronization method, which can determine the time difference between audio and video data, thereby performing synchronization processing on audio and video data to achieve lip-sync and improve the user's audio-visual experience.

[0064] The technical solutions provided by the embodiments of the present application can be applied to playback devices such as monitoring devices, live viewing devices, audio and video communication devices, and remote education and training devices that need to receive audio and video data in real time. Taking the playback device as an example, the technical solutions provided by the present application are described exemplarily.

[0065] The following combines Figures 5 to 8 The examples in describe the technical solutions of the embodiments of the present application.

[0066] As Figure 5As shown, a method for a playback device to determine the time difference between audio data and video data may include the following steps:

[0067] Step S501: Obtain the video sampling rate and the audio sampling rate.

[0068] In an embodiment of the present application, an electronic device may obtain the video sampling rate corresponding to the video data it receives and the audio sampling rate corresponding to the audio data. The video sampling rate and the audio sampling rate may be two different sampling rates. For example, the video sampling rate corresponding to the video data may be 90 kHz, and the audio sampling rate corresponding to the audio data may be 8 kHz, 16 kHz, 44.1 kHz, 32 kHz, 48 kHz, 64 kHz, etc. The present application does not make any limitations here.

[0069] Step S502: Determine the synchronization period according to the video sampling rate and the audio sampling rate.

[0070] In an embodiment of the present application, the synchronization period (also referred to as the mantissa period) may be a period in which the timestamps corresponding to the video data and the audio data start counting from 0 simultaneously after multiple rounds of overflow periods.

[0071] For example, Figure 6 is a schematic diagram of the overflow period of the audio and video timestamps provided by an embodiment of the present application. As Figure 6 shown, when the timestamp corresponding to the RTP data packet is 32 bits, and the video sampling rate is 90 kHz and the audio sampling rate is 8 kHz. The playback device can then determine that the audio timestamp overflow period is 149.1308 hours (approximately equal to 149.13 h), and the video overflow period is 13.256 hours (approximately equal to 13.26 h). Within one round of the overflow period of 149.1308 h (0x00000000 → 0xffffffff) of the audio and video timestamps, there are 11.25 rounds of video timestamp overflow periods.

[0072] It should be understood that during the first round of audio overflow cycle, the playback device can convert the timestamp of an audio frame in the audio data into the real audio time to calculate the real time of the corresponding video frame in the video data, so as to obtain the time difference between the audio data and the video data, and achieve the synchronous processing of the audio and video data. However, at the start time of the second round of audio timestamp overflow cycle, there is 0.75 video timestamp overflow cycle, and the playback device cannot directly calculate the real video time through the audio timestamp. At the start time of the third round of audio overflow cycle, there is 0.5 video overflow cycle. At the start time of the fourth round of audio overflow cycle, there is 0.25 video overflow cycle. At the start time of the fifth round of audio overflow cycle, it is also the start time of the video overflow cycle, and the playback device can also convert the timestamp of an audio frame in the audio data into the real audio time to calculate the real time of the corresponding video frame in the video data, so as to obtain the time difference between the audio data and the video data, and achieve the synchronous processing of the audio and video data. It can be understood that after four rounds of audio timestamp overflow cycles, the overflow cycles of the audio and video timestamps will start counting from 0 at the same time. That is, when the video sampling rate is 90kHz and the audio sampling rate is 8kHz, the audio and video synchronization cycle is 4.

[0073] For another example, Figure 7 is another schematic diagram of the audio and video timestamp overflow cycle provided by the embodiment of the present application. As Figure 7 shown, when the timestamp corresponding to the RTP data packet is 32 bits, and the video sampling rate is 90kHz and the audio sampling rate is 16kHz. The playback device can determine that there are 5.625 rounds of video timestamp overflow cycles within one round of overflow cycle of 74.57h of the audio timestamp through the ratio of the audio and video sampling periods 90000 / 16000 = 5.625. It should be understood that through eight rounds of overflow cycles of the audio timestamp, the overflow cycles of the audio and video timestamps will start counting from 0 at the same time. That is, when the video sampling rate is 90kHz and the audio sampling rate is 16kHz, the audio and video synchronization cycle is 8.

[0074] For another example, if the video sampling rate is 90kHz and the audio sampling rate is 44.1kHz, the playback device passes through the ratio of the audio and video sampling periods It can be seen that within one round of overflow cycle of the audio timestamp, there are 2 and 2 / 49 rounds of video timestamp overflow cycles. It can be understood that through 49 rounds of overflow cycles of the audio timestamp, the overflow cycles of the audio and video timestamps will start counting from 0 at the same time. That is, when the video sampling rate is 90kHz and the audio sampling rate is 44.1kHz, the audio and video synchronization cycle is 49.

[0075] In the embodiment of the present application, the method for determining the synchronization cycle according to the video sampling rate and the audio sampling rate may include:

[0076] Step A1: Determine the greatest common divisor of the video sampling rate and the audio sampling rate, which is the sampling rate common divisor.

[0077] The calculation formula for the sampling rate common divisor g can be:

[0078] g = gcd(Sa, Sv);

[0079] Wherein, Sa represents the audio sampling rate, with the unit of Hz; Sv represents the video sampling rate, with the unit of Hz.

[0080] For example, when the video sampling rate is 90 kHz and the audio sampling rate is 8 kHz, then g = gcd(8 kHz, 90 kHz) = 2 kHz.

[0081] Step A2: According to the sampling rate common divisor, respectively reduce the video sampling rate and the audio sampling rate to obtain the reduced video period and the reduced audio period.

[0082] The calculation formula for the reduced video period n can be:

[0083] n = Sv / g;

[0084] Wherein, Sv represents the video sampling rate, with the unit of Hz; g represents the sampling rate common divisor.

[0085] For example, when the video sampling rate is 90 kHz and the greatest common divisor is 2 kHz, the reduced video period n = 90 kHz / 2 kHz = 45.

[0086] The calculation formula for the reduced audio period m can be:

[0087] m = Sa / g;

[0088] Wherein, Sa represents the audio sampling rate, with the unit of Hz; g represents the sampling rate common divisor.

[0089] For example, when the audio sampling rate is 8 kHz and the greatest common divisor is 2 kHz, the reduced audio period m = 8 kHz / 2 kHz = 4.

[0090] Step A3: Determine the synchronization period according to the video sampling rate, the audio sampling rate, the reduced video period and the reduced audio period.

[0091] In the embodiment of the present application, if the video sampling rate is higher than the audio sampling rate, the synchronization period is the reduced audio period; if the audio sampling rate is higher than the video sampling rate, the synchronization period is the reduced video period.

[0092] Step S503: Determine the segmentation time according to the synchronization period.

[0093] In the embodiments of the present application, if the video sampling rate is higher than the audio sampling rate, the playback device may determine the segmentation time according to the video sampling rate and the synchronization period; if the audio sampling rate is higher than the video sampling rate, the playback device may determine the segmentation time according to the audio sampling rate and the synchronization period.

[0094] Exemplarily, if the video sampling rate is higher than the audio sampling rate, the method for the playback device to determine the segmentation time according to the video sampling rate and the synchronization period may include:

[0095] Step B1: Determine the video overflow period according to the video sampling rate and the number of bits of the timestamp.

[0096] In the embodiments of the present application, the number of bits of the timestamp is the number of bits of the timestamp corresponding to the RTP data packet, which is 32 bits.

[0097] The calculation formula for the playback device to determine the video overflow period according to the number of bits of the timestamp corresponding to the RTP data packet may be: where Sv represents the video sampling rate.

[0098] Step B2: Determine the ratio of the video overflow period to the synchronization period as the segmentation time.

[0099] The calculation formula for the segmentation time T may be where m represents the synchronization period.

[0100] It can be understood that if the audio sampling rate is higher than the video sampling rate, the method for the playback device to determine the segmentation time according to the audio sampling rate and the synchronization period may include:

[0101] Step C1: Determine the audio overflow period according to the audio sampling rate and the number of bits of the timestamp.

[0102] In the embodiments of the present application, the number of bits of the timestamp is the number of bits of the timestamp corresponding to the RTP data packet, which is 32 bits.

[0103] The calculation formula for the playback device to determine the audio overflow period according to the number of bits of the timestamp corresponding to the RTP data packet may be: where Sa represents the audio sampling rate.

[0104] Step C2: Determine the ratio of the audio overflow period to the synchronization period as the segmentation time.

[0105] The calculation formula for the segmentation time T may be where n represents the synchronization period.

[0106] According to the above B1 to B2, and steps C1 to C2, it can be seen that if the video sampling rate is higher than the audio sampling rate, the segmentation time If the audio sampling rate is higher than the video sampling rate, then Since m = Sa / g and n = Sv / g, it can be known that That is, the segmentation time is the product of the number of bits of the timestamp and the greatest common divisor of the sampling rates, divided by the product of the video sampling rate and the audio sampling rate.

[0107] Step S504: Obtain the video timestamp and the audio timestamp.

[0108] In the embodiment of the present application, the video timestamp can be the timestamp corresponding to any video frame in the video data received by the playback device, that is, the video timestamp is the timestamp in the RTP data packet where any video frame in the video data received by the playback device is located. The reception time of the audio frame corresponding to the audio timestamp and the video frame corresponding to the video timestamp can be the same time.

[0109] Step S505: Determine the relative video time according to the video timestamp, the video sampling rate, and the segmentation time.

[0110] In the embodiment of the present application, the relative video time V rel can be calculated by the formula:

[0111] V rel =(Tv / Sv) mod T;

[0112] wherein, Tv represents the video timestamp, Sv represents the video sampling rate, T represents the segmentation time, and (Tv / Sv) mod T represents the remainder of the ratio of the first ratio of the video timestamp Tv to the video sampling rate Sv to the segmentation time T.

[0113] Step S506: Determine the relative audio time according to the audio timestamp, the audio sampling rate, and the segmentation time.

[0114] In the embodiment of the present application, the relative audio time A rel can be calculated by the formula:

[0115] A rel =(Ta / Sa) mod T;

[0116] wherein, Ta represents the audio timestamp, Sa represents the audio sampling rate, T represents the segmentation time, and (Tv / Sv) mod T represents the remainder of the ratio of the second ratio of the audio timestamp Ta to the audio sampling rate Sa to the segmentation time T.

[0117] Step S507: Determine the first time difference according to the relative video time and the relative audio time.

[0118] In the embodiment of the present application, the first time difference d1 can be the difference between the relative audio time and the relative video time, that is, d1 = A rel - V rel .

[0119] Step S508: Determine a second time difference according to the first time difference and the segmentation time.

[0120] In the embodiment of the present application, the second time difference d2 can be the sum of the first time difference d1 and the segmentation time T, that is, d2 = d1 + T.

[0121] It can be understood that if the audio data is ahead of the video data, then the probability that the video data is in the previous overflow cycle (close to the overflow value) is relatively high, and the probability that the audio data is in the next overflow cycle (the timestamp is close to 0) is relatively high. The probability that the first time difference determined by the difference between the relative audio time and the relative video time is a very large negative value is relatively high. In the technical solution provided by the embodiment of the present application, by adding the segmentation time T on the basis of the first time difference, the situation where the calculation result is inaccurate caused by the audio data being ahead of the video data can be avoided.

[0122] Step S509: Determine a third time difference according to the first time difference and the segmentation time.

[0123] In the embodiment of the present application, the third time difference d3 can be the difference between the first time difference d1 and the segmentation time T, that is, d3 = d1 - T.

[0124] It can be understood that if the video data is ahead of the audio data, then the probability that the audio data is in the previous overflow cycle (close to the overflow value) is relatively high, and the probability that the video data is in the next overflow cycle (the timestamp is close to 0) is relatively high. The probability that the first time difference determined by the difference between the relative audio time and the relative video time is a very large positive value is relatively high. In the technical solution provided by the embodiment of the present application, by subtracting the segmentation time T on the basis of the first time difference, the situation where the calculation result is inaccurate caused by the video data being ahead of the audio data can be avoided.

[0125] Step S510: Determine a target time difference threshold according to the segmentation time.

[0126] In the embodiment of the present application, the method for the playback device to determine the target time difference threshold can be:

[0127] Step D1: Obtain a preset time threshold.

[0128] In the embodiment of the present application, the preset time threshold can be a time value preset by the user according to the usage scenario. For example, the preset time threshold T p can be 30 seconds, 25 seconds or 35 seconds, and the present application does not make any limitations here.

[0129] Step D2: Determine half of the segmentation time as the first time threshold.

[0130] The first time threshold can be the maximum allowable threshold of the audio-visual time difference. Exemplarily, if the segmentation time is T, then the first time threshold Tmax is T / 2.

[0131] Step D3: Determine a target time difference threshold according to a preset time threshold and a first time threshold, where the target time difference threshold is the minimum value of the preset time threshold and the first time threshold.

[0132] Exemplarily, the target time difference threshold T g can be calculated by the formula: T g = min(T max , T p );

[0133] where T max represents the first time threshold, T p represents the preset time threshold, and min(T max , T p ) represents taking the minimum value of the first time threshold and the preset time threshold.

[0134] In some embodiments, the target time difference threshold can also be the preset time threshold.

[0135] In some embodiments, the target time difference threshold can also be the first time threshold, or k times the segmentation time. For example, k can be 0.6, 0.8, 0.4, etc., which is not limited in this application.

[0136] Step S511: Determine a target time difference according to the target time difference threshold, the absolute value of the first time difference, the absolute value of the second time difference, and the absolute value of the third time difference.

[0137] In the embodiments of this application, after determining the first time difference, the second time difference, and the third time difference, absolute value operations can be performed on the first time difference, the second time difference, and the third time difference respectively to obtain the absolute values of the first time difference, the second time difference, and the third time difference. The target time difference can be one of the absolute values of the first time difference, the second time difference, and the third time difference that is less than the target time difference threshold.

[0138] If there is no time difference less than the target time difference threshold among the absolute values of the first time difference, the absolute value of the second time difference, and the absolute value of the third time difference, or if there are multiple time differences less than the target time difference threshold, it indicates that the starting counting moments of the audio data and the video data are inconsistent, or the time difference between the audio frames and the video frames themselves exceeds the alignment threshold, and the playback device does not perform synchronization processing on the audio and video data and can directly play the video data and the audio data.

[0139] In some embodiments, if the playback device supports the RTCP protocol and the packet format of the audio and video data is the RTCP packet format, the playback device can perform audio and video synchronization processing according to the additional timestamp information in the RTCP protocol, which is not elaborated in this application.

[0140] In the technical solution provided by the embodiment of the present application, the playback device can determine the audio-visual segmentation time according to the video sampling rate and the audio sampling rate. The segmentation time is the greatest common divisor of the video overflow period (video timestamp overflow period) corresponding to the video sampling rate and the audio overflow period (audio timestamp overflow period) corresponding to the audio sampling rate. Then, according to the segmentation time, the video sampling rate, the audio sampling rate, the video timestamp, and the audio timestamp, a plurality of time differences (including the first time difference, the second time difference, and the third time difference) are determined, and the target time difference is determined according to the absolute values of the plurality of time differences and the target time difference threshold. Thus, the audio-visual data is synchronously processed according to the target time difference to achieve lip-sync, thereby enhancing the user's audio-visual experience.

[0141] In the embodiment of the present application, the method for the playback device to synchronously process the audio-visual data according to the target time difference may include:

[0142] If the target time difference is positive, the playback of the second data to be synchronized is paused within the target time, and the target time is the time corresponding to the target time difference; if the target time difference is negative, the third data frame is played, and the third timestamp corresponding to the third data frame is the timestamp after the second timestamp and at an interval of the target time from the second timestamp, and the third data frame is one of the multiple second data frames.

[0143] It can be understood that if the target time difference is positive, it indicates that the current audio frame (second data frame) is ahead of the current video frame (first data frame), that is, the audio data is ahead of the video data. The playback device can then pause the playback of the audio data (cache the audio frame) within the target time and wait for the video data to play for the target time before continuing to play the audio data. If the target time difference is negative, it indicates that the current audio frame (second data frame) lags behind the current video frame (first data frame), that is, the audio data lags behind the video data. The playback device can then discard the video frame whose timestamp corresponds to the target time after the first data frame, and play the video frame that is separated from the first data frame by the target time after the first data frame.

[0144] If the target time difference is 0, it indicates that the probability that the current audio frame (second data frame) and the current video frame (first data frame) have no time difference is relatively high, that is, the audio data and the video data are in a synchronous playback state. The playback device does not need to synchronously process the audio data and the video data and directly plays the video data and the audio data.

[0145] In some embodiments, the present application also provides a data synchronization method, as Figure 8 shown, including the following steps:

[0146] Step S801: Obtain the first sampling rate corresponding to the first data to be synchronized, and obtain the second sampling rate corresponding to the second data to be synchronized. The first data to be synchronized includes multiple frames of first data, and the second data to be synchronized includes multiple frames of second data. The first sampling rate is higher than the second sampling rate.

[0147] In the embodiments of the present application, the first data to be synchronized and the second data to be synchronized can be audio data and video data with different sampling rates. Among them, the first data to be synchronized is video data, and the second data to be synchronized is audio data. Or, the first data to be synchronized is audio data, and the second data to be synchronized is video data. The present application does not make any limitations here.

[0148] In some embodiments, the first data to be synchronized and the second data to be synchronized can also be two video data with different sampling rates, or two audio data with different sampling rates. The present application does not make any limitations here.

[0149] In some embodiments, the first data to be synchronized and / or the second data to be synchronized can also be other data that needs to be transmitted in real time, such as sensor data (including pan-tilt angle information, acceleration information, etc.), detection box data for video detection (detection box position, detection box size, detection additional information: such as license plate number, body temperature, clothes color, etc.) or face recognition data (such as face feature information), etc. The present application does not make any limitations here.

[0150] Exemplarily, the first data to be synchronized can be face recognition data, and the second data to be synchronized can be detection box position data. The detection box position data can be used to represent the movement trajectory of a person, and the face recognition data can be used to assist in positioning the person's position.

[0151] Step S802: Determine the segmentation time according to the first sampling rate and the second sampling rate. The first overflow period corresponding to the first sampling rate is an integer multiple of the segmentation time, and the second overflow period corresponding to the second sampling rate is an integer multiple of the segmentation time.

[0152] In the embodiments of the present application, the method for the playback device to determine the segmentation time according to the first sampling rate and the second sampling rate may include: First, determine the greatest common divisor of the first sampling rate and the second sampling rate as the sampling rate common divisor. Then, determine the ratio of the second sampling rate to the sampling rate common divisor as the synchronization period. Finally, determine the ratio of the first overflow period to the synchronization period as the segmentation time.

[0153] The method for the playback device to determine the segmentation time according to the first sampling rate and the second sampling rate can refer to Figure 5 In the corresponding embodiments, the method for the playback device to determine the segmentation time according to the video sampling rate and the audio sampling rate will not be elaborated here in the present application.

[0154] Step S803: Determine the target time difference according to the segmentation time, the first sampling rate, the second sampling rate, the first timestamp, and the second timestamp. The first timestamp is the timestamp corresponding to the first data frame, the second timestamp is the timestamp corresponding to the second data frame, the first data frame is any one of multiple frames of first data, and the second data frame is the frame of data corresponding to the first data frame in multiple frames of second data.

[0155] In the embodiments of the present application, the method for the playback device to determine the target time difference according to the segmentation time, the first sampling rate, the second sampling rate, the first timestamp, and the second timestamp may include:

[0156] Step E1: Determine the first relative time according to the first timestamp, the first sampling rate, and the segmentation time.

[0157] Specifically, it includes: determining the ratio of the first timestamp to the first sampling rate as the first ratio. Then, determining the remainder of the ratio of the first ratio to the segmentation time as the first relative time.

[0158] Step E2: Determine the second relative time according to the second timestamp, the second sampling rate, and the segmentation time.

[0159] Specifically, it includes: determining the ratio of the second timestamp to the second sampling rate as the second ratio. Then, determining the remainder of the ratio of the second ratio to the segmentation time as the second relative time.

[0160] Step E3: Determine the first time difference according to the first relative time and the second relative time.

[0161] The first time difference is the difference between the second relative time and the first relative time.

[0162] Step E4: Determine the second time difference and the third time difference according to the first time difference and the segmentation time.

[0163] The second time difference is the sum of the first time difference and the segmentation time. The third time difference is the difference between the first time difference and the segmentation time.

[0164] Step E5: Determine the target time difference threshold according to the segmentation time.

[0165] Specifically, it includes: obtaining the preset time threshold and determining half of the segmentation time as the first time threshold. The target time difference threshold is the minimum value between the preset time threshold and the first time threshold.

[0166] Step E6: Determine the target time difference according to the target time difference threshold, the absolute value of the first time difference, the absolute value of the second time difference, and the absolute value of the third time difference.

[0167] The target time difference is the one among the absolute value of the first time difference, the absolute value of the second time difference, and the absolute value of the third time difference that is less than the target time difference threshold.

[0168] The method for a playback device to determine a target time difference according to a segmentation time, a first sampling rate, a second sampling rate, a first timestamp, and a second timestamp can be referred to Figure 5 In the corresponding embodiment, for the method of the playback device to determine the target time difference according to the segmentation time, video sampling rate, audio sampling rate, video timestamp, and audio timestamp, this application will not elaborate here.

[0169] Step S804: Synchronize the first data to be synchronized and the second data to be synchronized according to the target time difference.

[0170] If the target time difference is positive, pause playing the second data to be synchronized within the target time, where the target time is the time corresponding to the target time difference. If the target time difference is negative, play the third data frame, and the third timestamp corresponding to the third data frame is the timestamp after the second timestamp and separated from the second timestamp by the target time, and the third data frame is one of multiple frames of the second data.

[0171] In the technical solution provided by the embodiments of the present application, the playback device can obtain the first sampling rate corresponding to the first data to be synchronized and obtain the second sampling rate corresponding to the second data to be synchronized. The first data to be synchronized includes multiple frames of first data, the second data to be synchronized includes multiple frames of second data, and the first sampling rate is higher than the second sampling rate. Then, according to the first sampling rate and the second sampling rate, determine the segmentation time, where the first overflow period corresponding to the first sampling rate is an integer multiple of the segmentation time, and the second overflow period corresponding to the second sampling rate is an integer multiple of the segmentation time. Then, according to the segmentation time, the first sampling rate, the second sampling rate, the first timestamp, and the second timestamp, determine the target time difference, where the first timestamp is the timestamp corresponding to the first data frame, the second timestamp is the timestamp corresponding to the second data frame, the first data frame is any one of multiple frames of the first data, and the second data frame is the frame of the second data corresponding to the first data frame. Finally, synchronize the first data to be synchronized and the second data to be synchronized according to the target time difference. In the technical solution provided by the embodiments of the present application, by analyzing the target time difference, the playback device can accurately adjust the synchronization relationship between the first data to be synchronized and the second data to be synchronized, and finally achieve audio-visual synchronization, significantly improving the user's audio-visual experience. The technical solution provided by the embodiments of the present application ensures the accuracy and reliability of the synchronization process through reasonable sampling rate analysis and timestamp calculation.

[0172] It should be understood that, on the premise of no logical conflict, the above-mentioned various application embodiments can be combined and implemented with each other to meet the actual application requirements. The specific embodiments or implementation schemes obtained after these combinations still fall within the protection scope of this application.

[0173] Corresponding to the data synchronization method in the above embodiments, an embodiment of the present application provides a data synchronization device. The data synchronization device can be implemented by software, hardware, or a combination of both to form part or all of a computer device, and is used to execute the steps in the data synchronization method in the above embodiments.

[0174] Figure 9 FIG. 4 shows a schematic structural diagram of a data synchronization device 90 provided by an embodiment of the present application. For ease of description, only parts related to the embodiment of the present application are shown.

[0175] Referring to Figure 9 , the device 90 includes an acquisition module 910 and a processing module 920.

[0176] The acquisition module 910 is configured to acquire a first sampling rate corresponding to first data to be synchronized, and acquire a second sampling rate corresponding to second data to be synchronized. The first data to be synchronized includes multiple frames of first data, the second data to be synchronized includes multiple frames of second data, and the first sampling rate is higher than the second sampling rate.

[0177] The processing module 920 is configured to determine a segmentation time according to the first sampling rate and the second sampling rate. A first overflow period corresponding to the first sampling rate is an integer multiple of the segmentation time, and a second overflow period corresponding to the second sampling rate is an integer multiple of the segmentation time.

[0178] The processing module 920 is further configured to determine a target time difference according to the segmentation time, the first sampling rate, the second sampling rate, a first timestamp, and a second timestamp. The first timestamp is a timestamp corresponding to a first data frame, the second timestamp is a timestamp corresponding to a second data frame, the first data frame is any one of the multiple frames of first data, and the second data frame is a frame of data corresponding to the first data frame in the multiple frames of second data.

[0179] The processing module 920 is further configured to perform synchronization processing on the first data to be synchronized and the second data to be synchronized according to the target time difference.

[0180] In some embodiments, the processing module 920 is specifically configured to: determine a first relative time according to the first timestamp, the first sampling rate, and the segmentation time. Determine a second relative time according to the second timestamp, the second sampling rate, and the segmentation time. Determine a first time difference according to the first relative time and the second relative time. Determine a second time difference and a third time difference according to the first time difference and the segmentation time. Acquire a target time difference threshold. Determine the target time difference according to the target time difference threshold, the absolute value of the first time difference, the absolute value of the second time difference, and the absolute value of the third time difference.

[0181] In some embodiments, the processing module 920 is specifically configured to: determine the ratio of the first timestamp to the first sampling rate as the first ratio. Determine the remainder of the division of the first ratio by the segmentation time as the first relative time. Determine the second relative time according to the second timestamp, the second sampling rate, and the segmentation time, including: determine the ratio of the second timestamp to the second sampling rate as the second ratio. Determine the remainder of the division of the second ratio by the segmentation time as the second relative time.

[0182] In some embodiments, the processing module 920 is specifically configured to: determine the difference between the second relative time and the first relative time as the first time difference. Determine the sum of the first time difference and the segmentation time as the second time difference. Determine the difference between the first time difference and the segmentation time as the third time difference.

[0183] In some embodiments, the processing module 920 is specifically configured to: obtain a preset time threshold. Determine half of the segmentation time as the first time threshold. Determine the target time difference threshold according to the preset time threshold and the first time threshold, where the target time difference threshold is the minimum of the preset time threshold and the first time threshold.

[0184] In some embodiments, the target time difference is one of the absolute values of the first time difference, the absolute value of the second time difference, and the absolute value of the third time difference that is less than the target time difference threshold.

[0185] In some embodiments, the processing module 920 is specifically configured to: determine the greatest common divisor of the first sampling rate and the second sampling rate as the sampling rate common divisor. Determine the ratio of the second sampling rate to the sampling rate common divisor as the synchronization period. Determine the ratio of the first overflow period to the synchronization period as the segmentation time.

[0186] In some embodiments, the processing module 920 is specifically configured to: if the target time difference is positive, pause playing the second data to be synchronized within the target time, where the target time is the time corresponding to the target time difference. If the target time difference is negative, play the third data frame, and the third timestamp corresponding to the third data frame is the timestamp after the second timestamp and separated from the second timestamp by the target time, and the third data frame is one of multiple frames of the second data.

[0187] In some embodiments, the first data to be synchronized is video data, and the first sampling rate is the video sampling rate. The second data to be synchronized is audio data, and the second sampling rate is the audio sampling rate.

[0188] It should be noted that for the information interaction, execution process, etc. between the above-mentioned device / units, since they are based on the same concept as the method embodiments of the present application, for their specific functions and the technical effects brought, reference can be specifically made to the method embodiment part, and details are not described herein again.

[0189] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0190] Based on the same inventive concept, an embodiment of the present application also provides an electronic device.

[0191] Figure 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 10 shown, the electronic device 10 of this embodiment includes: at least one processor 1010 ( Figure 10 only one is shown in the figure), a memory 1020, and a communication module 1040. A computer program 1030 that may run on the processor 1010 is stored in the memory 1020. When the processor 1010 executes the computer program 1030, it implements the steps in the above-mentioned data synchronization method embodiment, such as Figure 5 steps 501 to 511 shown in the figure. Or Figure 8 steps 801 to 804 shown in the figure. When the processor 1010 executes the computer program 1030, it implements the functions of each module / unit in the above-mentioned device embodiments, such as Figure 9 the functions of module 910 to module 920 shown in the figure. The communication module 1040 may be a separate communication unit for communicating with an external server or a terminal device.

[0192] The electronic device 10 may include, but is not limited to: a processor 1010 and a memory 1020. Those skilled in the art can understand that Figure 10 this is only an example of the electronic device 10 and does not constitute a limitation to the electronic device 10. It may include more or fewer components than those shown in the figure, or combine some components, or different components. For example, the electronic device 10 may also include an input and sending device, a network access device, a bus, etc.

[0193] The processor 1010 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0194] The memory 1020 can be an internal storage unit of the electronic device 10 in some embodiments, such as the hard disk or memory of the electronic device 10. The memory 1020 can also be an external storage device of the electronic device 10, such as a plug-in hard disk equipped on the electronic device 10, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. The memory 1020 can also include both the internal storage unit of the electronic device 10 and an external storage device. The memory 1020 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of the computer program 1030. The memory 1020 can also be used to temporarily store data that has been sent or will be sent.

[0195] In addition, those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. In each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0196] The embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program runs on an electronic device, the electronic device is caused to execute the steps in the above-mentioned method embodiments.

[0197] The embodiments of the present application provide a chip. The chip includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the steps in the above-mentioned method embodiments are implemented.

[0198] The embodiments of the present application provide a computer program product. When the computer program product runs on an electronic device, the electronic device is caused to execute the steps in the above-mentioned method embodiments.

[0199] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0200] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0201] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0202] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0203] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in the form of hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0204] In the embodiments provided in this application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, in each embodiment of the present application, each functional unit can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0207] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the large-screen device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0208] Finally, it should be noted that the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Acquire a first sampling rate corresponding to first data to be synchronized, and acquire a second sampling rate corresponding to second data to be synchronized, wherein the first data to be synchronized includes multiple frames of first data, the second data to be synchronized includes multiple frames of second data, and the first sampling rate is higher than the second sampling rate; Determine a segmentation time according to the first sampling rate and the second sampling rate, wherein a first overflow period corresponding to the first sampling rate is an integer multiple of the segmentation time, and a second overflow period corresponding to the second sampling rate is an integer multiple of the segmentation time; Determine a target time difference according to the segmentation time, the first sampling rate, the second sampling rate, a first timestamp and a second timestamp, wherein the first timestamp is a timestamp corresponding to a first data frame, the second timestamp is a timestamp corresponding to a second data frame, the first data frame is any one frame of data in the multiple frames of first data, and the second data frame is a frame of data in the multiple frames of second data corresponding to the first data frame; The first data to be synchronized and the second data to be synchronized are synchronized according to the target time difference.

2. The data synchronization method according to claim 1, characterized in that: The step of determining the target time difference according to the segmentation time, the first sampling rate, the second sampling rate, the first timestamp and the second timestamp includes: Determining a first relative time according to the first timestamp, the first sampling rate, and the segmentation time; Determine a second relative time according to the second timestamp, the second sampling rate and the split time; determining a first time difference according to the first relative time and the second relative time; Determine a second time difference and a third time difference according to the first time difference and the split time; Get the target time difference threshold; The target time difference is determined according to the target time difference threshold, the first time difference, the second time difference and the third time difference.

3. The data synchronization method according to claim 2, characterized in that: The determining the first relative time according to the first timestamp, the first sampling rate and the segmentation time includes: Determine a ratio of the first timestamp to the first sampling rate as a first ratio; determining a remainder of a ratio of the first ratio to the split time as the first relative time; The determining the second relative time according to the second timestamp, the second sampling rate and the segmentation time includes: Determine a ratio of the second timestamp to the second sampling rate as a second ratio; The remainder of the ratio of the second ratio to the split time is determined as the second relative time.

4. The data synchronization method according to claim 2, characterized in that: The determining a first time difference according to the first relative time and the second relative time includes: Determine a difference between the second relative time and the first relative time as the first time difference; The determining the second time difference and the third time difference according to the first time difference and the split time includes: Determine the sum of the first time difference and the split time as the second time difference; The difference between the first time difference and the split time is determined as the third time difference.

5. The data synchronization method according to claim 2, characterized in that: The step of obtaining the target time difference threshold comprises: Get the preset time threshold; Determine half of the split time as the first time threshold; The target time difference threshold is determined according to the preset time threshold and the first time threshold, and the target time difference threshold is the minimum value between the preset time threshold and the first time threshold.

6. The data synchronization method according to claim 2, characterized in that: The target time difference is one of the absolute value of the first time difference, the absolute value of the second time difference, and the absolute value of the third time difference, which is smaller than the target time difference threshold.

7. The data synchronization method according to claim 1, characterized in that: The step of determining the segmentation time according to the first sampling rate and the second sampling rate includes: Determine the greatest common divisor of the first sampling rate and the second sampling rate as the sampling rate common divisor; Determine a ratio of the second sampling rate to a common divisor of the sampling rates as a synchronization period; The ratio of the first overflow period to the synchronization period is determined as the division time.

8. The data synchronization method according to claim 1, characterized in that: The step of synchronizing the first data to be synchronized and the second data to be synchronized according to the target time difference includes: If the target time difference is a positive number, the playing of the second data to be synchronized is paused within the target time, and the target time is the time corresponding to the target time difference; If the target time difference is a negative number, the third data frame is played. The third timestamp corresponding to the third data frame is a timestamp after the second timestamp and separated from the second timestamp by the target time. The third data frame is a frame of data in the multiple frames of second data.

9. The data synchronization method according to any one of claims 1 to 8, characterized in that: The first data to be synchronized is video data, and the first sampling rate is a video sampling rate; The second data to be synchronized is audio data, and the second sampling rate is an audio sampling rate.

10. An electronic device, characterized in that: The device comprises a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the data synchronization method as described in any one of claims 1 to 9.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data synchronization method as described in any one of claims 1 to 9 is implemented.

Images