Audio and video synchronization method, apparatus and device, and readable storage medium

Abstract

The invention discloses an audio and video synchronization method, device and equipment and a readable storage medium. The audio and video synchronization method comprises the following steps: acquiring an acquired audio; determining audio abrupt change points of the audio playing device and the video playing device in the collected audio; on the basis of moments corresponding to the audio abrupt change points of the audio playing device and the video playing device and a second preset duration, calculating to obtain a delay duration; and if the delay duration exceeds the preset delay range, adjusting the video playing speed of the video playing device according to the delay duration so as to synchronize audios and videos of the audio playing device and the video playing device. According to the invention, the collected audio is the superposed audio of the characteristic audio played by the audio playing device and the video playing device, so that the audio mutation points corresponding to the audio playing device and the video playing device can be easily distinguished, and the delay duration between the audio playing device and the video playing device can be accurately calculated. Therefore, the audio and the video can be played synchronously by adjusting the video playing speed.

Description

Technical Field

[0001] The present application relates to the field of audio and video processing technology, and in particular to an audio and video synchronization method, device, equipment and readable storage medium. Background Art

[0002] With the widespread popularity of digital TV and home entertainment systems, set-top boxes have gradually become one of the indispensable devices in the home. Set-top boxes are usually connected to TVs via HDMI (High Definition Multimedia Interface) to receive and play various video content, such as live TV and video on demand. In order to enhance the user's viewing experience, more and more users choose to use external audio devices (such as Bluetooth speakers) instead of built-in speakers on TVs to obtain better sound quality. However, the problem of asynchronous audio and video playback also arises.

[0003] In a typical home entertainment environment, the set-top box is responsible for decoding and outputting audio and video data. The video is transmitted to the TV through the HDMI interface, while the audio can be transmitted to the external speaker via Bluetooth. Since the audio and video data are played on different devices, and these devices have different processing capabilities, network conditions, Bluetooth connection quality and other factors, it is easy to cause the audio and video playback to be out of sync, giving users a poor audio-visual experience. In summary, there is a problem of playback asynchrony between the video played on the TV connected to the set-top box and the audio played on the Bluetooth speaker connected to the set-top box, which gives users a poor audio-visual experience. Summary of the invention

[0004] The present application provides an audio and video synchronization method, apparatus, device and readable storage medium, aiming to solve the technical problem that there is a playback asynchrony problem between the video played by a TV connected to a set-top box and the audio played by a Bluetooth speaker connected to the set-top box, which brings a poor audio-visual experience to users.

[0005] In a first aspect, an embodiment of the present application provides an audio and video synchronization method, the audio and video synchronization method comprising:

[0006] Acquire collected audio, where the collected audio is collected from audio played by an audio playback device and a video playback device, wherein the audio playback device plays a first characteristic audio in a loop at intervals of a first preset duration, and the video playback device plays a second characteristic audio in a loop at intervals of the first preset duration, and the video playback device starts playing after the audio playback device starts playing and after the second preset duration, the first preset duration is greater than the second preset duration, and the first preset duration and the second preset duration are used to avoid overlap between the first characteristic audio and the second characteristic audio;

[0007] Determine the audio mutation points of the audio playback device and the video playback device in the collected audio;

[0008] Calculate the delay duration based on the time corresponding to the audio mutation point of the audio playback device and the video playback device and the second preset duration;

[0009] If the delay duration exceeds the preset delay range, the video playback speed of the video playback device is adjusted according to the delay duration to synchronize the audio and video of the audio playback device and the video playback device.

[0010] Optionally, determining the audio mutation points of the audio playback device and the video playback device in the collected audio includes:

[0011] Preprocessing the collected audio to obtain processed audio data;

[0012] Based on the processed audio data, a preset number of data points whose data value change rates exceed a preset change rate are selected in a sampling order to obtain an audio mutation point sequence, wherein the preset number is an even number;

[0013] The odd-numbered audio mutation points in the audio mutation point sequence are used as the audio mutation points of the audio playback device, and the even-numbered audio mutation points are used as the audio mutation points of the video playback device.

[0014] Optionally, the preprocessing of the collected audio to obtain processed audio data includes:

[0015] The collected audio is decoded to obtain raw audio data, the raw audio data is converted from a stereo format to a mono format, and noise in the raw audio data is removed to obtain processed audio data.

[0016] Optionally, the calculating of the delay duration based on the moment corresponding to the audio mutation point of the audio playback device and the video playback device and the second preset duration includes:

[0017] Combining an audio mutation point of an audio playback device and an audio mutation point of a video playback device together according to a sampling order to obtain a plurality of mutation point combinations;

[0018] For each mutation point combination, the time corresponding to the audio mutation point of the video playback device is subtracted from the time corresponding to the audio mutation point of the audio playback device to obtain the difference of each mutation point combination;

[0019] Take the average of the differences of all mutation point combinations to get the delay difference;

[0020] The delay duration is obtained by subtracting the second preset duration from the delay difference.

[0021] Optionally, before obtaining the difference of each mutation point combination by subtracting the time corresponding to the audio mutation point of the audio playback device from the time corresponding to the audio mutation point of the video playback device for each mutation point combination, the method further includes:

[0022] The sampling sequence number of each audio mutation point is divided by the sampling frequency to obtain the time corresponding to each audio mutation point.

[0023] Optionally, if the delay duration exceeds a preset delay range, adjusting the video playback speed of the video playback device according to the delay duration includes:

[0024] If the delay duration exceeds the preset delay range and the delay duration is a positive value, the video playback speed of the video playback device is increased, wherein the reduced duration obtained by increasing the video playback speed is equal to the absolute value of the delay duration.

[0025] Optionally, if the delay duration exceeds a preset delay range, adjusting the video playback speed of the video playback device according to the delay duration further includes:

[0026] If the delay duration exceeds the preset delay range and is a negative value, the video playback speed of the video playback device is reduced, wherein the increased duration obtained by reducing the video playback speed is equal to the absolute value of the delay duration.

[0027] In a second aspect, an embodiment of the present application provides an audio and video synchronization device, the audio and video synchronization device comprising:

[0028] An acquisition module is used to acquire collected audio, wherein the collected audio is collected from audio played by an audio playback device and a video playback device, wherein the audio playback device plays a first characteristic audio in a loop at intervals of a first preset time length, and the video playback device plays a second characteristic audio in a loop at intervals of the first preset time length, and the video playback device starts playing after the audio playback device starts playing after a second preset time length, wherein the first preset time length is greater than the second preset time length, and the first preset time length and the second preset time length are used to avoid overlap between the first characteristic audio and the second characteristic audio;

[0029] A determination module, used to determine the audio mutation points of the audio playback device and the video playback device in the collected audio;

[0030] A calculation module, used to calculate the delay duration based on the time corresponding to the audio mutation point of the audio playback device and the video playback device and the second preset duration;

[0031] The adjustment module is used to adjust the video playback speed of the video playback device according to the delay duration if the delay duration exceeds a preset delay range, so as to synchronize the audio and video of the audio playback device and the video playback device.

[0032] In a third aspect, an embodiment of the present application provides an audio and video synchronization device, which includes a processor, a memory, and an audio and video synchronization program stored in the memory and executable by the processor, wherein when the audio and video synchronization program is executed by the processor, the steps of the audio and video synchronization method described above are implemented.

[0033] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which an audio and video synchronization program is stored, wherein when the audio and video synchronization program is executed by a processor, the steps of the audio and video synchronization method as described above are implemented.

[0034] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0035] In an embodiment of the present application, collected audio is obtained, and the collected audio is collected from the audio played by the audio playback device and the video playback device, wherein the audio playback device plays the first characteristic audio in a loop at intervals of a first preset duration, and the video playback device plays the second characteristic audio in a loop at intervals of the first preset duration, and the video playback device starts playing after the audio playback device starts playing for a second preset duration, and the first preset duration is greater than the second preset duration and the first preset duration and the second preset duration are used to avoid overlap between the first characteristic audio and the second characteristic audio; determine the audio mutation point of the audio playback device and the video playback device in the collected audio; calculate the delay duration based on the time corresponding to the audio mutation point of the audio playback device and the video playback device and the second preset duration; if the delay duration exceeds the preset delay range, adjust the video playback speed of the video playback device according to the delay duration to synchronize the audio and video of the audio playback device and the video playback device. In an embodiment of the present application, the collected audio is a superimposed audio of the characteristic audios played by the audio playback device and the video playback device respectively, wherein the playback periods of the first characteristic audio and the second characteristic audio are the same, and the characteristics may be the same or different, and the video playback device starts playing after a second preset time period after the audio playback device starts playing, and the first preset time period is greater than the second preset time period, so that the first characteristic audio and the second characteristic audio can be played alternately and orderly, and the first preset time period and the second preset time period are set to avoid overlapping of the first characteristic audio and the second characteristic audio, so that for the collected superimposed audio, the audio mutation points corresponding to the audio playback device and the video playback device can be easily distinguished, and then based on the time corresponding to the audio mutation points of the two combined with the second preset time period, the delay time between the audio playback device and the video playback device can be accurately calculated, so that the audio and video playback can be synchronized by adjusting the video playback speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of an embodiment of the audio and video synchronization method of the present application;

[0037] Figure 2 A schematic diagram of the collected audio spectrum of an embodiment of the audio and video synchronization method of the present application;

[0038] Figure 3 For this application Figure 1 Detailed flow chart of step S20;

[0039] Figure 4 For this application Figure 1 Detailed flow chart of step S30;

[0040] Figure 5 A schematic diagram of an audio and video synchronization adjustment process of an embodiment of an audio and video synchronization method of the present application;

[0041] Figure 6 This is a schematic diagram of the system architecture of an embodiment of the audio and video synchronization method of the present application;

[0042] Figure 7 This is a schematic diagram of the remote control side process of an embodiment of the audio and video synchronization method of the present application;

[0043] Figure 8 This is a schematic diagram of a characteristic audio playback process of an embodiment of the audio and video synchronization method of the present application;

[0044] Fig. 9 A schematic diagram of the audio receiving and communication process of an embodiment of the audio and video synchronization method of the present application;

[0045] Fig.10 This is a functional module diagram of an embodiment of an audio and video synchronization device of the present application;

[0046] Fig.11 This is a schematic diagram of the hardware structure of the audio and video synchronization device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0049] In a first aspect, an embodiment of the present application provides an audio and video synchronization method.

[0050] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the audio and video synchronization method of the present application. Figure 1 As shown, the audio and video synchronization method includes:

[0051] Step S10, obtaining collected audio, wherein the collected audio is collected from the audio played by the audio playback device and the video playback device, wherein the audio playback device plays the first characteristic audio in a loop at intervals of a first preset time length, and the video playback device plays the second characteristic audio in a loop at intervals of a first preset time length, and the video playback device starts playing after the audio playback device starts playing after a second preset time length, wherein the first preset time length is greater than the second preset time length and the first preset time length and the second preset time length are used to avoid overlapping of the first characteristic audio and the second characteristic audio.

[0052] In this embodiment, the collected audio is first obtained. The collected audio can be collected by an audio collection device to collect the audio played by the audio playback device and the video playback device. Figure 2 , Figure 2 This is a schematic diagram of the collected audio spectrum of an embodiment of the audio and video synchronization method of the present application. It is easy to understand that the collected audio is actually a superimposed audio formed by collecting the respective characteristic audios played by the audio playback device and the video playback device. The playback period of the first characteristic audio and the second characteristic audio are the same, and the characteristics can be the same or different. To facilitate data analysis, the first characteristic audio and the second characteristic audio can choose to play a characteristic audio with a short duration. For example, the first characteristic audio can choose a characteristic audio with a "beep" sound, and the second characteristic audio can change the timbre, pitch and volume of the first characteristic audio based on the first characteristic audio, or use more audios with different characteristics. This is not limited here. The first The preset duration is greater than the second preset duration, so that the first characteristic audio and the second characteristic audio can be played alternately and orderly, and the first characteristic audio and the second characteristic audio can be prevented from overlapping by setting the first preset duration and the second preset duration, that is, the first characteristic audio and the second characteristic audio can be prevented from being played simultaneously at the same time. Specifically, the audio playback device plays the first characteristic audio "beep" in a loop at an interval of the first preset duration, such as 2 seconds, and the video playback device starts playing after the audio playback device starts playing after a second preset duration, such as 500 milliseconds. The video playback device also plays the second characteristic audio in a loop at an interval of the first preset duration, such as 2 seconds, so that the collected superimposed audio appears as follows. Figure 2 The two frequency spectrum features shown are cross-ordered and non-overlapping, which makes it easy to analyze the collected superimposed audio.

[0053] Step S20, determining the audio mutation points of the audio playback device and the video playback device in the collected audio.

[0054] In this embodiment, by analyzing the collected audio data, the audio mutation points of the audio playback device and the video playback device in the collected audio are determined, and the audio mutation points of the audio playback device and the video playback device in the collected audio are determined. Figure 2 , Figure 2 The position pointed by each slash in is the audio mutation point, where the audio mutation points of the audio playback device and the video playback device are calculated according to Figure 2 The order from left to right shown appears alternately, that is, Figure 2 The audio mutation points pointed to by the 1st, 3rd, 5th, 7th, 9th, 11th and 13th oblique lines are the audio mutation points of the audio playback device. Figure 2 The audio mutation points pointed to by the 2nd, 4th, 6th, 8th, 10th, 12th and 14th oblique lines are the audio mutation points of the video playback device. It is easy to understand that the audio mutation points are the data points where the audio data values ​​change significantly.

[0055] Step S30, based on the time corresponding to the audio mutation point of the audio playback device and the video playback device and the second preset time, the delay time is calculated.

[0056] In this embodiment, continue to refer to Figure 2 , since it is known Figure 2 Therefore, for example, the delay duration can be calculated by combining the moment corresponding to the audio mutation point of the first audio playback device and the first video playback device with the second preset duration.

[0057] Step S40: If the delay duration exceeds the preset delay range, the video playback speed of the video playback device is adjusted according to the delay duration to synchronize the audio and video of the audio playback device and the video playback device.

[0058] In this embodiment, an acceptable preset delay range can be pre-set. If the delay duration does not exceed the preset delay range, there is no need to adjust the audio and video synchronization. If the delay duration exceeds the preset delay range, the video playback speed of the video playback device is adjusted according to the delay duration, so that the audio and video of the audio playback device and the video playback device are synchronized. Since the human ear is highly sensitive to sound, the playback speed of the audio device is not adjusted, but the video playback speed of the video playback device is adjusted, thereby reducing the user's perception of the audio and video synchronization adjustment and improving the user experience.

[0059] In this embodiment, firstly, collected audio is obtained. The collected audio can be used to collect the audio played by the audio playback device and the video playback device through an audio collection device. The collected audio is the superimposed audio of the characteristic audio played by the audio playback device and the video playback device respectively, wherein the playback period of the first characteristic audio and the second characteristic audio are the same, and the characteristics may be the same or different. After the audio playback device starts playing, the video playback device starts playing after a second preset time. The first preset time is greater than the second preset time, which can make the first characteristic audio and the second characteristic audio be played crosswise and orderly, and the setting of the first preset time and the second preset time can avoid the first characteristic audio and the second characteristic audio from being played alternately and orderly. The two characteristic audios overlap, so that the audio mutation points corresponding to the audio playback device and the video playback device can be easily distinguished from the collected superimposed audio, and then based on the moments corresponding to the audio mutation points of the two combined with the second preset duration, the delay duration between the audio playback device and the video playback device can be accurately calculated, and an acceptable preset delay range can be set in advance. If the delay duration does not exceed the preset delay range, there is no need to adjust the audio and video synchronization. If the delay duration exceeds the preset delay range, the video playback speed of the video playback device is adjusted according to the delay duration, so that the audio and video of the audio playback device and the video playback device are synchronized.

[0060] Further, in one embodiment, referring to Figure 3 , Figure 3 For this application Figure 1 The detailed flow chart of step S20 is as follows: Figure 3 As shown, step S20 includes:

[0061] Step S201, preprocessing the collected audio to obtain processed audio data;

[0062] Step S202, based on the processed audio data, selecting a preset number of data points whose data value change rate exceeds a preset change rate in a sampling order to obtain an audio mutation point sequence, wherein the preset number is an even number;

[0063] Step S203: taking the odd-numbered audio mutation points in the audio mutation point sequence as the audio mutation points of the audio playback device, and taking the even-numbered audio mutation points as the audio mutation points of the video playback device.

[0064] In this embodiment, the collected audio can first be decoded and pre-processed to remove noise to obtain processed audio data. The audio mutation point is the data point where the audio data value changes significantly. Figure 2 , Figure 2The position pointed by each slash in is the audio mutation point. Since the first characteristic audio is played before the second characteristic audio, the first preset duration is greater than the second preset duration, and the first preset duration and the second preset duration are set to avoid the first characteristic audio and the second characteristic audio from overlapping. Therefore, it is easy to conclude that the audio mutation points of the audio playback device and the video playback device will be based on Figure 2 The order from left to right shown appears alternately, that is, Figure 2 The audio mutation points pointed to by the 1st, 3rd, 5th, 7th, 9th, 11th and 13th oblique lines are the audio mutation points of the audio playback device. Figure 2 The audio mutation points pointed to by the 2nd, 4th, 6th, 8th, 10th, 12th and 14th oblique lines are the audio mutation points of the video playback device.

[0065] Furthermore, in one embodiment, the preprocessing of the collected audio to obtain processed audio data includes:

[0066] The collected audio is decoded to obtain raw audio data, the raw audio data is converted from a stereo format to a mono format, and noise in the raw audio data is removed to obtain processed audio data.

[0067] In this embodiment, if the collected audio is in a coded format, the collected audio is decoded to obtain raw audio data. If the raw audio data is in a stereo format, the stereo format is converted into a mono format, and the noise in the raw audio data is removed to obtain processed audio data, so as to improve the efficiency and accuracy of subsequent data processing.

[0068] Further, in one embodiment, referring to Figure 4 , Figure 4 For this application Figure 1 The detailed flow chart of step S30 is as follows: Figure 4 As shown, step S30 includes:

[0069] Step S301, combining an audio mutation point of an audio playback device and an audio mutation point of a video playback device in a sampling order to obtain a plurality of mutation point combinations;

[0070] Step S302, for each mutation point combination, subtract the time corresponding to the audio mutation point of the audio playback device from the time corresponding to the audio mutation point of the video playback device to obtain a difference value for each mutation point combination;

[0071] Step S303, taking the average of the differences of all mutation point combinations to obtain a delay difference;

[0072] Step S304: subtract the second preset duration from the delay difference to obtain the delay duration.

[0073] In this embodiment, continue to refer to Figure 2 , Figure 2 There are 14 audio mutation points in total, among which the 1st, 3rd, 5th, 7th, 9th, 11th and 13th audio mutation points are audio mutation points of the audio playback device, and the 2nd, 4th, 6th, 8th, 10th, 12th and 14th audio mutation points are audio mutation points of the video playback device. The 1st and 2nd, 3rd and 4th, 5th and 6th, 7th and 8th, 9th and 10th, 11th and 12th, 13th and 14th audio mutation points are respectively combined together to obtain 7 groups of mutation point combinations. For each mutation point combination, for example, for the combination of the 1st and 2nd audio mutation points, the time corresponding to the audio mutation point of the first audio playback device is subtracted from the time corresponding to the audio mutation point of the second video playback device to obtain the difference of the mutation point combination, the differences of all the 7 groups of mutation point combinations are averaged to obtain the delay difference, and the delay duration is obtained by subtracting the second preset duration from the delay difference. It should be noted that the number of mutation point combinations can be set according to specific needs. In the process of averaging the differences of all mutation point combinations, outliers can be removed to improve the accuracy of the delay duration.

[0074] Furthermore, in one embodiment, before step S302, the following steps are included:

[0075] The sampling sequence number of each audio mutation point is divided by the sampling frequency to obtain the time corresponding to each audio mutation point.

[0076] In this embodiment, the sampling number of the audio mutation point is used to record the sampling number of the audio mutation point in all data points. The unit of the sampling frequency is, for example, the number of data points collected per second. Therefore, the sampling number of each audio mutation point is divided by the sampling frequency to calculate the time corresponding to each audio mutation point.

[0077] Furthermore, in one embodiment, step S40 includes:

[0078] If the delay duration exceeds the preset delay range and the delay duration is a positive value, the video playback speed of the video playback device is increased, wherein the reduced duration obtained by increasing the video playback speed is equal to the absolute value of the delay duration.

[0079] In this embodiment, refer to Figure 5 , Figure 5 FIG. 1 is a schematic diagram of an audio and video synchronization adjustment process according to an embodiment of the audio and video synchronization method of the present application. Figure 5 As shown, if the delay duration exceeds the preset delay range and the delay duration is a positive value, it means that there is a delay lag between the video playback device and the audio playback device, then the video playback speed of the video playback device is increased, and the reduced duration obtained by increasing the video playback speed is equal to the absolute value of the delay duration, so that the playback progress of the video playback device catches up with the playback progress of the audio playback device.

[0080] Furthermore, in one embodiment, step S40 further includes:

[0081] If the delay duration exceeds the preset delay range and is a negative value, the video playback speed of the video playback device is reduced, wherein the increased duration obtained by reducing the video playback speed is equal to the absolute value of the delay duration.

[0082] In this embodiment, continue to refer to Figure 5 If the delay duration exceeds the preset delay range and is a negative value, it indicates that the audio playback device is delayed relative to the video playback device. The video playback speed of the video playback device is reduced. The increased duration obtained by reducing the video playback speed is equal to the absolute value of the delay duration, so that the playback progress of the audio playback device catches up with the playback progress of the video playback device.

[0083] Further, in one embodiment, referring to Figure 6 , Figure 6 This is a schematic diagram of the system architecture of an embodiment of the audio and video synchronization method of the present application. Figure 6 As shown, the audio and video synchronization device is a set-top box, the audio playback device is an external speaker, the video playback device is a television, and the audio acquisition device is a set-top box remote controller. Before step S10, the following steps are included:

[0084] The set-top box controls the set-top box remote controller to start the audio collection function and collect the audio played by the external speakers and the TV.

[0085] In this embodiment, the audio and video synchronization device is a set-top box, and the set-top box includes but is not limited to various devices with audio and video playback functions such as IPTV set-top boxes, OTT set-top boxes and projection boxes. The audio playback device is an external speaker, including but not limited to wired speakers, speakers using a local area network, Bluetooth speakers and other audio playback devices. The video playback device is a television, including but not limited to traditional non-smart TVs, smart TVs, monitors, projectors and other video playback devices. The audio acquisition device is a set-top box remote control, including but not limited to a voice remote control using the Huawei Star Flash protocol, a voice remote control using a dongle wireless transmitter, and a mobile phone with a set-top box control function and an audio acquisition function. In addition, the audio acquisition device can also be a wired microphone, a wireless microphone and other independent devices with audio acquisition functions. Reference Figure 7 , Figure 7 This is a schematic diagram of the remote control side process of an embodiment of the audio and video synchronization method of the present application, such as Figure 7As shown, before step S10, the set-top box establishes a communication connection with the remote control by pairing the remote control with the Bluetooth protocol, the remote control receives the microphone-on instruction sent by the set-top box, activates the microphone on the remote control to start collecting audio, encodes the collected audio and sends it to the set-top box, and after the audio and video synchronization adjustment is completed, the remote control receives the microphone-off instruction sent by the set-top box by the Bluetooth protocol, switches the remote control to the non-pickup state and turns off the audio collection. Figure 8 , Figure 8 This is a characteristic audio playback process diagram of an embodiment of the audio and video synchronization method of the present application, such as Figure 8 As shown, the set-top box first reads the first characteristic audio and the second characteristic audio, and then controls the external speaker to play the first characteristic audio through Bluetooth, and controls the TV to play the second characteristic audio through the HDMI interface. Under normal circumstances, after the set-top box is connected to an external audio playback device (such as a Bluetooth speaker), the HDMI interface will not transmit sound, but only transmit video data. By modifying the Android audio routing logic, different audio streams can be routed to different output devices, so that multiple devices can play different sounds at the same time. The embodiment of the present application can achieve the adaptation of set-top boxes, TVs and external audio devices of different brands and models without adding hardware devices or modifying existing devices, and maintain good versatility. The embodiment of the present application creatively converts the problem of measuring TV video delay into measuring TV audio delay, cleverly avoiding the impact of the delay between Bluetooth remote control and set-top box communication on the result, and actively communicating with the Bluetooth remote control, solves the problem of collecting audio on microphone-free devices such as set-top boxes. The embodiment of the present application is not only applicable to traditional TV programs and movie playback, but also can be extended to other multimedia content, such as online live broadcast, video conferencing, games and other scenes, to ensure the audio and video synchronization effect of multimedia content. In addition, the embodiments of the present application are not only applicable to the scenario of a single set-top box, a TV, and a Bluetooth speaker, but can also be extended to the synchronous application of multiple devices to ensure the audio and video synchronization effect when playing on multiple devices. Fig. 9 , Fig. 9 Schematic diagram of the audio receiving and communication process of an embodiment of the audio and video synchronization method of the present application, such as Fig. 9As shown, the audio acquisition device can use the device's built-in microphone or an external microphone. It is usually equipped with a remote control with voice recognition, which can control the remote control to collect and transmit sound. Before playing, the microphone-on command is sent to the Bluetooth voice remote control through the Bluetooth protocol. The remote control will switch to the sound pickup state and return the collected audio data to the set-top box device, and then determine whether the audio encoding format is PCM raw stream or audio encoded in SBC, ADPCM and other formats. If it is encoded audio, an audio decoder is required to decode it into the original PCM format. After the synchronization is completed, the microphone-off command is sent to the Bluetooth voice remote control through the Bluetooth protocol to switch the remote control to the non-pickup state.

[0086] Furthermore, in one embodiment, after step S40, the following steps are included:

[0087] Return to the step of obtaining the collected audio.

[0088] In this embodiment, in order to ensure the audio and video synchronization effect, after the audio and video synchronization adjustment in step S40, return to execute step S10 again, and perform multiple audio and video synchronization adjustments until the calculated delay duration is within the preset delay range, then the audio and video synchronization adjustment is completed. It should be noted that after the audio and video synchronization adjustment is completed, the delay duration can be stored in the set-top box. When subsequently playing on-demand or live broadcast services, the underlying module of the set-top box player automatically calls the stored delay duration to ensure the synchronization effect during the audio and video playback process.

[0089] Furthermore, in one embodiment, obtaining the collected audio includes:

[0090] When it is detected that the audio playback device and / or the video playback device is replaced, the captured audio is obtained.

[0091] In this embodiment, when it is detected that the audio playback device and / or video playback device has been replaced, the collected audio is obtained, that is, the audio and video synchronization detection and adjustment process is entered to ensure the audio and video synchronization effect of the replaced audio playback device and / or video playback device.

[0092] Furthermore, in one embodiment, the audio and video synchronization method further includes:

[0093] Provide audio and video synchronization settings through the operation interface.

[0094] In this embodiment, the user can monitor the audio and video synchronization status on the operation interface and manually trigger synchronization adjustment when necessary, thereby further improving the flexibility and user controllability of synchronization adjustment. During the audio and video synchronization adjustment process, the user can obtain real-time feedback information on the synchronization adjustment through the display interface or audio and video prompts to understand the synchronization progress and status.

[0095] In a second aspect, an embodiment of the present application also provides an audio and video synchronization device.

[0096] In one embodiment, referring to Fig.10 , Fig.10 This is a functional module diagram of an embodiment of the audio and video synchronization device of the present application. Fig.10 As shown, the audio and video synchronization device includes:

[0097] An acquisition module 10 is used to acquire collected audio, wherein the collected audio is collected from audio played by an audio playback device and a video playback device, wherein the audio playback device plays a first characteristic audio in a loop at intervals of a first preset duration, and the video playback device plays a second characteristic audio in a loop at intervals of a first preset duration, and the video playback device starts playing after the audio playback device starts playing after a second preset duration, wherein the first preset duration is greater than the second preset duration, and the first preset duration and the second preset duration are used to avoid overlap between the first characteristic audio and the second characteristic audio;

[0098] A determination module 20, used to determine the audio mutation points of the audio playback device and the video playback device in the collected audio;

[0099] A calculation module 30, configured to calculate a delay duration based on the time corresponding to the audio mutation point of the audio playback device and the video playback device and a second preset duration;

[0100] The adjustment module 40 is used to adjust the video playback speed of the video playback device according to the delay duration if the delay duration exceeds a preset delay range, so as to synchronize the audio and video of the audio playback device and the video playback device.

[0101] Furthermore, in one embodiment, the determination module 20 is used to:

[0102] Preprocessing the collected audio to obtain processed audio data;

[0103] Based on the processed audio data, a preset number of data points whose data value change rates exceed a preset change rate are selected in a sampling order to obtain an audio mutation point sequence, wherein the preset number is an even number;

[0104] The odd-numbered audio mutation points in the audio mutation point sequence are used as the audio mutation points of the audio playback device, and the even-numbered audio mutation points are used as the audio mutation points of the video playback device.

[0105] Furthermore, in one embodiment, the calculation module 30 is used to:

[0106] Combining an audio mutation point of an audio playback device and an audio mutation point of a video playback device together according to a sampling order to obtain a plurality of mutation point combinations;

[0107] For each mutation point combination, the time corresponding to the audio mutation point of the video playback device is subtracted from the time corresponding to the audio mutation point of the audio playback device to obtain the difference of each mutation point combination;

[0108] Take the average of the differences of all mutation point combinations to get the delay difference;

[0109] The delay duration is obtained by subtracting the second preset duration from the delay difference.

[0110] Furthermore, in one embodiment, the audio and video synchronization device further includes a preprocessing module, which is used to:

[0111] The collected audio is decoded to obtain raw audio data, the raw audio data is converted from a stereo format to a mono format, and noise in the raw audio data is removed to obtain processed audio data.

[0112] Furthermore, in one embodiment, the audio and video synchronization device further includes a time determination module, which is used to:

[0113] The sampling sequence number of each audio mutation point is divided by the sampling frequency to obtain the time corresponding to each audio mutation point.

[0114] Furthermore, in one embodiment, the adjustment module 40 is further configured to:

[0115] If the delay duration exceeds the preset delay range and the delay duration is a positive value, the video playback speed of the video playback device is increased, wherein the reduced duration obtained by increasing the video playback speed is equal to the absolute value of the delay duration.

[0116] Furthermore, in one embodiment, the adjustment module 40 is further configured to:

[0117] If the delay duration exceeds the preset delay range and is a negative value, the video playback speed of the video playback device is reduced, wherein the increased duration obtained by reducing the video playback speed is equal to the absolute value of the delay duration.

[0118] Among them, the functional implementation of each module in the above-mentioned audio and video synchronization device corresponds to the various steps in the above-mentioned audio and video synchronization method embodiment, and its functions and implementation processes are no longer repeated here.

[0119] In a third aspect, an embodiment of the present application provides an audio and video synchronization device.

[0120] Reference Fig.11 , Fig.11 The hardware structure diagram of the audio and video synchronization device involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the audio and video synchronization device may include a processor, a memory, a communication interface, and a communication bus.

[0121] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0122] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the audio and video synchronization device, and the interface used to realize the interconnection between the audio and video synchronization device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0123] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0124] The processor may be a general-purpose processor, and the general-purpose processor may call the audio and video synchronization program stored in the memory and execute the audio and video synchronization method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the audio and video synchronization program is called may refer to the various embodiments of the audio and video synchronization method of the present application, and will not be repeated here.

[0125] Those skilled in the art will understand that Fig.11 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0126] In a fourth aspect, an embodiment of the present application also provides a readable storage medium.

[0127] An audio and video synchronization program is stored on the readable storage medium of the present application, wherein when the audio and video synchronization program is executed by the processor, the steps of the audio and video synchronization method as described above are implemented.

[0128] Among them, the method implemented when the audio and video synchronization program is executed can refer to the various embodiments of the audio and video synchronization method of the present application, and will not be repeated here.

[0129] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0130] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0131] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0132] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0133] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0135] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An audio and video synchronization method, characterized in that: The audio and video synchronization method comprises: Acquire collected audio, where the collected audio is collected from audio played by an audio playback device and a video playback device, wherein the audio playback device plays a first characteristic audio in a loop at intervals of a first preset duration, and the video playback device plays a second characteristic audio in a loop at intervals of the first preset duration, and the video playback device starts playing after the audio playback device starts playing and after the second preset duration, the first preset duration is greater than the second preset duration, and the first preset duration and the second preset duration are used to avoid overlap between the first characteristic audio and the second characteristic audio; Determine the audio mutation points of the audio playback device and the video playback device in the collected audio; Calculate the delay duration based on the time corresponding to the audio mutation point of the audio playback device and the video playback device and the second preset duration; If the delay duration exceeds the preset delay range, the video playback speed of the video playback device is adjusted according to the delay duration to synchronize the audio and video of the audio playback device and the video playback device.

2. The audio and video synchronization method according to claim 1, characterized in that: Determining the audio mutation points of the audio playback device and the video playback device in the collected audio includes: Preprocessing the collected audio to obtain processed audio data; Based on the processed audio data, a preset number of data points whose data value change rates exceed a preset change rate are selected in a sampling order to obtain an audio mutation point sequence, wherein the preset number is an even number; The odd-numbered audio mutation points in the audio mutation point sequence are used as the audio mutation points of the audio playback device, and the even-numbered audio mutation points are used as the audio mutation points of the video playback device.

3. The audio and video synchronization method according to claim 2, characterized in that: The audio data obtained by preprocessing the collected audio includes: The collected audio is decoded to obtain raw audio data, the raw audio data is converted from a stereo format to a mono format, and noise in the raw audio data is removed to obtain processed audio data.

4. The audio and video synchronization method according to claim 2, characterized in that: The calculating of the delay duration based on the time corresponding to the audio mutation point of the audio playback device and the video playback device and the second preset duration includes: Combining an audio mutation point of an audio playback device and an audio mutation point of a video playback device together according to a sampling order to obtain a plurality of mutation point combinations; For each mutation point combination, the time corresponding to the audio mutation point of the video playback device is subtracted from the time corresponding to the audio mutation point of the audio playback device to obtain the difference of each mutation point combination; Take the average of the differences of all mutation point combinations to get the delay difference; The delay duration is obtained by subtracting the second preset duration from the delay difference.

5. The audio and video synchronization method according to claim 4, characterized in that: Before obtaining the difference of each mutation point combination by subtracting the time corresponding to the audio mutation point of the audio playback device from the time corresponding to the audio mutation point of the video playback device for each mutation point combination, the method includes: The sampling sequence number of each audio mutation point is divided by the sampling frequency to obtain the time corresponding to each audio mutation point.

6. The audio and video synchronization method according to claim 1, characterized in that: If the delay duration exceeds the preset delay range, adjusting the video playback speed of the video playback device according to the delay duration includes: If the delay duration exceeds the preset delay range and the delay duration is a positive value, the video playback speed of the video playback device is increased, wherein the reduced duration obtained by increasing the video playback speed is equal to the absolute value of the delay duration.

7. The audio and video synchronization method according to claim 1, characterized in that: If the delay duration exceeds the preset delay range, adjusting the video playback speed of the video playback device according to the delay duration also includes: If the delay duration exceeds the preset delay range and is a negative value, the video playback speed of the video playback device is reduced, wherein the increased duration obtained by reducing the video playback speed is equal to the absolute value of the delay duration.

8. An audio and video synchronization device, characterized in that: The audio and video synchronization device comprises: An acquisition module is used to acquire collected audio, wherein the collected audio is collected from audio played by an audio playback device and a video playback device, wherein the audio playback device plays a first characteristic audio in a loop at intervals of a first preset time length, and the video playback device plays a second characteristic audio in a loop at intervals of the first preset time length, and the video playback device starts playing after the audio playback device starts playing after a second preset time length, wherein the first preset time length is greater than the second preset time length, and the first preset time length and the second preset time length are used to avoid overlap between the first characteristic audio and the second characteristic audio; A determination module, used to determine the audio mutation points of the audio playback device and the video playback device in the collected audio; A calculation module, used to calculate the delay duration based on the time corresponding to the audio mutation point of the audio playback device and the video playback device and the second preset duration; The adjustment module is used to adjust the video playback speed of the video playback device according to the delay duration if the delay duration exceeds a preset delay range, so as to synchronize the audio and video of the audio playback device and the video playback device.

9. An audio and video synchronization device, characterized in that: The audio and video synchronization device includes a processor, a memory, and an audio and video synchronization program stored in the memory and executable by the processor, wherein when the audio and video synchronization program is executed by the processor, the steps of the audio and video synchronization method as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: An audio and video synchronization program is stored on the readable storage medium, and when the audio and video synchronization program is executed by the processor, the steps of the audio and video synchronization method according to any one of claims 1 to 7 are implemented.

Images