Audio and video synchronization method and device, medium and equipment
By processing audio signals in the transmitting device and generating a second audio signal of ultrasonic information, it is transmitted to the receiving device to synchronize video and sound, and the time difference problem of synchronous playback of video and sound in a multi-screen interactive environment is solved, which significantly improves users audio-visual experience.
Patent Information
- Application Number
- CN202411204615.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-15
AI Technical Summary
In vehicle space or multimedia scenarios, there is a time difference in synchronous playback of video and sound in multi-screen interactive environment, resulting in poor user audio-visual experience. There are errors and delays in the transmission of estimation time information through the network, resulting in the playback progress of video and sound. Inconsistent.
In the transmitting device, the audio signal to be played is processed to generate a second audio signal including ultrasonic information, and plays it through the sound player. The receiving device determines the current playback time from the ultrasonic information of the sound signal to play the video synchronously.
Real-time playback time is obtained through ultrasonic transmission, reducing estimation errors and network transmission delay errors, significantly improving the synchronization between video and sound, and improving the user's audio-visual experience.
Smart Images

Figure CN118972655A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to signal processing technology, and in particular to a method, device, medium and equipment for synchronizing audio and video. Background Art
[0002] In the vehicle space or other multimedia scenes, there is often a need for multi-screen interaction. For example, in the vehicle space, there may be multiple independent video playback devices. The playback of the sound corresponding to the video can be controlled by one of the target video playback devices (called the sending end device). The other video playback devices (called the receiving end device) except the target video playback device share a set of sound playback devices with the target video playback device. In the case where multiple users need to watch videos through multiple video playback devices, in order to ensure the synchronization of the video and sound watched by each user, the relevant technology usually transmits the estimated time information of the audio signal to be played to other video playback devices through network transmission by the sound playback control device. The other video playback devices adjust the playback position of the played video according to the received time information in the hope of achieving synchronization between the video and the sound. However, since the transmitted time information is estimated, there is a certain error with the actual playback time information, and the network transmission process will cause a certain time delay. The superposition of the two errors leads to a large time difference between the video playback progress and the sound playback progress of other video playback devices. This time difference often enables users to perceive the asynchrony between the video and the sound, resulting in a poor user audio-visual experience. Summary of the invention
[0003] The embodiments of the present disclosure provide a method, apparatus, medium and device for synchronizing audio and video, which can improve the synchronization of video and sound, thereby enhancing the user's audio-visual experience.
[0004] According to a first aspect of an embodiment of the present disclosure, a method for synchronizing audio and video is provided for a transmitting device, the method comprising: determining a first audio signal to be played; processing the first audio signal to generate a second audio signal comprising ultrasonic information; the ultrasonic information comprising time information corresponding to the first audio signal; playing a sound signal corresponding to the second audio signal, so that a receiving device determines a current playback time from the ultrasonic information of the sound signal, and synchronously plays a corresponding video according to the current playback time.
[0005] A second aspect of an embodiment of the present disclosure provides a method for synchronizing audio and video for a receiving device, the method comprising: collecting a sound signal played by a sending device to obtain an audio signal corresponding to the sound signal; in response to the audio signal including ultrasonic information, determining a current playback time corresponding to the sound signal from the ultrasonic information of the audio signal; and synchronously playing the corresponding video according to the current playback time.
[0006] According to a third aspect of an embodiment of the present disclosure, there is provided an audio and video synchronization device for a transmitting device, the device comprising: a first determination module for determining a first audio signal to be played; a first processing module for processing the first audio signal to generate a second audio signal comprising ultrasonic information; the ultrasonic information comprises time information corresponding to the first audio signal; and a playback control module for playing a sound signal corresponding to the second audio signal, so that the receiving device determines the current playback time from the ultrasonic information of the sound signal, and synchronously plays the corresponding video according to the current playback time.
[0007] According to a fourth aspect of the embodiments of the present disclosure, there is provided an audio and video synchronization device for a receiving device, the device comprising: a signal acquisition module, for acquiring a sound signal played by a sending device, and obtaining an audio signal corresponding to the sound signal; a second processing module, for determining a current playback time corresponding to the sound signal from the ultrasonic information of the audio signal in response to the audio signal including ultrasonic information; and a synchronization control module, for synchronously playing the corresponding video according to the current playback time.
[0008] A fifth aspect of the embodiments of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the audio and video synchronization method described in any of the above embodiments of the present disclosure.
[0009] The sixth aspect of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the audio and video synchronization method described in any of the above embodiments of the present disclosure.
[0010] A seventh aspect of the present disclosure provides a computer program product. When instructions in the computer program product are executed by a processor, the audio and video synchronization method provided in any of the above embodiments of the present disclosure is performed.
[0011] Based on the audio and video synchronization method, device, medium and equipment provided by the above embodiments of the present disclosure, at the sending end device, the first audio signal to be played can be determined; the first audio signal is processed to generate a second audio signal including ultrasonic information; the sound signal corresponding to the second audio signal is played, so that the receiving end device can collect the sound signal corresponding to the played second audio signal. Since the second audio signal includes ultrasonic information, and the ultrasonic information includes time information corresponding to the first audio signal, the receiving end device can obtain the current playback time of the real-time played sound signal from the ultrasonic information of the sound signal, and then the corresponding video can be played synchronously according to the current playback time, so as to achieve the synchronization of the video played by the receiving end device and the sound played by the sending end device. And the real-time playback time of the sound signal obtained by the receiving end device through ultrasonic transmission can effectively reduce the estimation error and the network transmission delay error compared with the estimation time through network transmission in the related art, so as to greatly improve the synchronization of the video playback progress of the receiving end device and the sound playback progress of the sending end device, and effectively improve the user's audio-visual experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 This is an exemplary application scenario of the audio and video synchronization method provided by the present disclosure;
[0013] Figure 2 is a flowchart of a method for synchronizing audio and video provided by an exemplary embodiment of the present disclosure;
[0014] Figure 3 is a flowchart of a method for synchronizing audio and video provided by another exemplary embodiment of the present disclosure;
[0015] Figure 4 is a flowchart of a method for synchronizing audio and video provided by another exemplary embodiment of the present disclosure;
[0016] Figure 5 is a flowchart of a method for synchronizing audio and video provided by another exemplary embodiment of the present disclosure;
[0017] Figure 6 is a flowchart of a method for synchronizing audio and video provided by an exemplary embodiment of the present disclosure;
[0018] Figure 7 is a structural schematic diagram of an audio and video synchronization device provided by an exemplary embodiment of the present disclosure;
[0019] Figure 8 is a structural schematic diagram of an audio and video synchronization device provided by another exemplary embodiment of the present disclosure;
[0020] Fig. 9is a structural schematic diagram of an audio and video synchronization device provided by another exemplary embodiment of the present disclosure;
[0021] Fig.10 is a structural schematic diagram of an audio and video synchronization device provided by yet another exemplary embodiment of the present disclosure;
[0022] Fig.11 is a structural diagram of an audio and video synchronization system provided by an exemplary embodiment of the present disclosure;
[0023] Fig.12 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0024] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.
[0025] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0026] SUMMARY OF THE DISCLOSURE
[0027] In the process of implementing the present disclosure, the inventors found that in the vehicle space or other multimedia scenes, there is often a demand for multi-screen interaction. For example, in the vehicle space, there may be multiple independent video playback devices, and the playback of the sound corresponding to the video can be controlled by one of the target video playback devices (called the sending end device). Other video playback devices (called receiving end devices) except the target video playback device share a set of sound playback devices with the target video playback device. When multiple users need to watch videos through multiple video playback devices, in order to ensure the synchronization of the video and sound watched by each user, the relevant technology usually transmits the estimated time information of the audio signal to be played to other video playback devices through the network by the sound playback control device. The other video playback devices adjust the playback position of the played video according to the received time information, in the hope of achieving synchronization between the video and the sound. However, since the transmission time information is estimated, there is a certain error between it and the actual playback time information, and the network transmission process will cause a certain time delay. The two errors are superimposed, resulting in a large time difference between the video playback progress and the sound playback progress of other video playback devices. For example, the time difference is 1 second or more. This time difference often enables users to perceive the synchronization between video and sound, resulting in a poor user audio-visual experience.
[0028] Exemplary Overview
[0029] Figure 1 This is an exemplary application scenario of the audio and video synchronization method provided by the present disclosure. Figure 1 As shown, in the vehicle space scene, a transmitting end device 12 is provided in the vehicle 11, and the transmitting end device 12 in the figure takes the video playback device (video playback device can also be called video player) of the front row as an example. One or more receiving end devices 14 can be provided on the backrest of the front seat 13. The receiving end device 14 can be, for example, a video playback device watched by the user on the back seat 15. The transmitting end device 12 can control the in-vehicle sound player 16 to play the sound corresponding to the video. The transmitting end device 12 and the receiving end device 14 can use the audio and video synchronization method of the embodiment of the present disclosure to achieve synchronous playback of video and audio. Specifically, the transmitting end device 12 can determine the first audio signal to be played; the transmitting end device 12 processes the first audio signal to generate a second audio signal including ultrasonic information; the ultrasonic information includes time information corresponding to the first audio signal; the transmitting end device 12 can play the sound signal corresponding to the second audio signal through the in-vehicle sound player 16, and the receiving end device 14 can collect the sound signal played by the sound player 16, and obtain the current playback time of the played sound signal from the ultrasonic information of the sound signal, and then play the corresponding video synchronously according to the current playback time. The transmission of the real-time playback time of the sound is realized through ultrasound, so as to synchronize the video and sound played by the receiving device 14. Compared with the estimation time through network transmission in the related art, the embodiment of the present disclosure can effectively reduce the estimation error and the network transmission delay error, thereby greatly improving the synchronization of the video playback progress and the sound playback progress of the receiving device 14. In practical applications, the playback of sound is not limited to the in-car sound player 16 set in the figure, but can also be any other implementable sound playback device (also called sound player). For example, the sending device 12 can be configured with a sound playback device, and the sound signal can be directly played through the sending device.
[0030] The audio and video synchronization method of the disclosed embodiment is not limited to application in vehicle space scenarios, but can also be applied to any other multimedia scenarios that require videos played by different video playback devices to be synchronized with the same sound.
[0031] Exemplary Methods
[0032] Figure 2 FIG. 1 is a flow chart of a method for synchronizing audio and video provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to an electronic device, such as a car computer, and the electronic device serves as a transmitting device. Figure 2 As shown, the method of the embodiment of the present disclosure may include the following steps:
[0033] Step 210: determine a first audio signal to be played.
[0034] Among them, the first audio signal to be played can be an audio signal corresponding to the video to be played by the transmitting end device. The first audio signal is an electrical signal. For example, in the vehicle space, when the front-row user watches a video through the front-row video player, the first audio signal to be played is the audio signal corresponding to the video to be played by the front-row video player. The video to be played by the transmitting end device can be determined according to the viewing needs of the user. For example, the front-row user selects the video to be played through the front-row video player, and the video to be played and the corresponding first audio signal are determined according to the user's selection.
[0035] In some optional embodiments, the first audio signal may be an audio signal of a partial duration or the entire duration of the video to be played. For example, if the entire duration of the video to be played is 2 hours, the first audio signal may be an audio signal within the 10-minute duration of the current video to be played, or may be an audio signal of 2 hours duration. That is, according to the preset duration, each audio signal segment of the preset duration in the complete audio signal of the video to be played may be determined as the first audio signal to be played in sequence. Alternatively, the complete audio signal may be determined as the first audio signal to be played.
[0036] In some optional embodiments, the first audio signal can be obtained from a device that provides a video, and the video can be stored in a video file. The device that provides the video can be, for example, a server corresponding to a video application or a terminal device that runs a video application, or, if it is a local video of a transmitting device or a video pre-downloaded from a server, the first audio signal can be obtained from a local corresponding storage area. The manner of obtaining the first audio signal and the video is not limited.
[0037] Step 220: Process the first audio signal to generate a second audio signal including ultrasonic information; the ultrasonic information includes time information corresponding to the first audio signal.
[0038] The processing of the first audio signal may include adding an ultrasonic frequency band to the first audio signal, modulating ultrasonic information, etc., so that the ultrasonic information can be added to the first audio signal to generate a second audio signal including the ultrasonic information. The ultrasonic information includes time information corresponding to the first audio signal. That is, the time information corresponding to the first audio signal is modulated into ultrasonic information, added to the first audio signal, and the second audio signal is obtained.
[0039] In some optional embodiments, the time information corresponding to the first audio signal may include a timestamp corresponding to at least one audio sampling point in the first audio signal.
[0040] In some optional embodiments, the ultrasonic information may be coded modulation information or non-coded modulation information, without specific limitation. Coding refers to the technology of compressing audio signals according to a specified coding method, which is used to reduce data redundancy in audio to improve compression rate and transmission efficiency.
[0041] Step 230, play the sound signal corresponding to the second audio signal, so that the receiving end device determines the current play time from the ultrasonic information of the sound signal, and synchronously plays the corresponding video according to the current play time.
[0042] In some optional embodiments, the second audio signal is an electrical signal, and the sound signal corresponding to the second audio signal is a sound wave signal converted from the second audio signal. The sound signal corresponding to the second audio signal can be played by a sound player, that is, the second audio signal of the electrical signal is converted into a sound signal by the sound player so as to propagate in the air. For example, in the case where the transmitting end device itself includes a sound player, the sound signal corresponding to the second audio signal can be played by the sound player of the transmitting end device. In the case where the transmitting end device itself does not include a sound player, the sound signal corresponding to the second audio signal can be played by a sound player independent of the transmitting end device and connected to the transmitting end device.
[0043] In some optional embodiments, the receiving device may be an electronic device that is in the same multimedia space (e.g., in-vehicle space) as the transmitting device and is independent of the receiving device. The electronic device can play (or play through a video player connected to the electronic device) a video. For example, in the in-vehicle space, a rear-row video player device that is independent of the front-row vehicle head unit and the front-row video player device. Or a rear-row vehicle head unit that is connected to the rear-row video player device and can control the rear-row video player device to play a video.
[0044] In some optional embodiments, the receiving device can collect the sound signal corresponding to the second audio signal through a specific audio collection device (or sound collection device, sound collector, audio collector, etc.) to obtain an audio signal that is the same as or similar to the second audio signal. The audio signal similar to the second audio signal may have a different sampling rate, but the audio content contained in the sampled audio signal is the same. Then, the receiving device can determine the time information of the currently played audio signal from the ultrasonic information of the audio signal, and determine the time information of the audio signal as the current playback time. The receiving device can synchronously play the corresponding video according to the current playback time.
[0045] In some optional embodiments, when the receiving device can obtain the video corresponding to the first audio signal, the receiving device can play the corresponding video, and when the current playback time of the sound signal is obtained, the progress of the played video can be controlled according to the current playback time, so that the video played by the receiving device is synchronized with the sound signal controlled to be played by the sending device.
[0046] In some optional embodiments, when a first user watches a video through a first video playback device corresponding to a transmitting device, and a second user watches a video through a second video playback device corresponding to a receiving device, the receiving device can determine the current playback time from the ultrasonic information of the sound signal, and adjust the playback speed of the second video playback device according to the current playback time, so as to continuously reduce the time difference between the video played by the second video playback device and the current playback time, thereby gradually synchronizing the video played by the second video playback device with the sound signal. Alternatively, the receiving device can control the second video playback device to start playing from the video frame corresponding to the current playback time according to the current playback time, so that the video played by the second video playback device is synchronized with the sound signal. The synchronization method of the receiving device synchronizing the corresponding video according to the current playback time is not limited.
[0047] In the audio and video synchronization method provided in this embodiment, since the second audio signal includes ultrasonic information, and the ultrasonic information includes time information corresponding to the first audio signal, the receiving device can obtain the current playback time of the real-time played sound signal from the ultrasonic information of the sound signal corresponding to the second audio signal, and then can play the corresponding video synchronously according to the current playback time, so as to achieve synchronization between the video played by the receiving device and the sound played by the sending device. Moreover, the real-time playback time of the sound signal obtained by the receiving device through ultrasonic transmission can effectively reduce the estimation error and network transmission delay error compared to the time estimated through network transmission in the related art, thereby greatly improving the synchronization between the video playback progress of the receiving device and the sound playback progress of the sending device.
[0048] Figure 3 It is a flowchart of a method for synchronizing audio and video provided by another exemplary embodiment of the present disclosure.
[0049] In some optional embodiments, in the above Figure 2 Based on the embodiment shown, Figure 3 As shown, step 220 processes the first audio signal to generate a second audio signal including ultrasonic information, which may specifically include the following steps:
[0050] Step 2210: modulate and generate ultrasonic information according to the time information corresponding to the first audio signal.
[0051] In some optional embodiments, any feasible modulation method may be used to modulate and generate ultrasonic information.
[0052] In some optional embodiments, the time information corresponding to the first audio signal can be converted into any codable form to obtain encoded time information, for example, the time information can be converted into binary or other digital codes to facilitate modulation of the time information into ultrasonic information.
[0053] In some optional embodiments, an ultrasonic transducer may be used to modulate the encoded time information onto an ultrasonic signal to obtain modulated ultrasonic information. The modulation method may include, for example, frequency modulation, phase modulation (or amplitude modulation), etc. That is, the ultrasonic signal may be able to represent the encoded time information by changing the characteristics of the ultrasonic signal such as the frequency, phase (or amplitude). The ultrasonic information may include information such as the frequency, phase (or amplitude) of the ultrasonic wave.
[0054] Step 2220: generate a second audio signal by mixing the first audio signal and the ultrasonic information.
[0055] Mixing may refer to adding (or superimposing) ultrasonic information to the first audio signal to obtain a second audio signal including the ultrasonic information.
[0056] In some optional embodiments, an ultrasonic part can be added to a position on the first audio signal corresponding to the ultrasonic information, and modulated ultrasonic information can be added to the ultrasonic part to obtain a second audio signal including the ultrasonic information. During the playback of the second audio signal, the ultrasonic information is played and transmitted along with the playback of the second audio signal. The ultrasonic information will not be perceived by the user, but can be collected by an audio collection device of a receiving device that is capable of collecting ultrasonic waves. The receiving device can then obtain time information from the ultrasonic information as the current playback time for synchronous playback of video and sound.
[0057] In this embodiment, ultrasonic information is generated by modulating the time information corresponding to the first audio signal, and the first audio signal is mixed with the ultrasonic information to generate a second audio signal, so that the ultrasonic information can be transmitted to the receiving device through the playback of the second audio signal and collected by the receiving device, so that the receiving device can obtain the real-time playback time of the sound, and provide the receiving device with an accurate and effective current playback time, thereby improving the synchronization between the video played by the receiving device and the sound controlled to be played by the sending device.
[0058] In some optional embodiments, the time information corresponding to the first audio signal may include a timestamp corresponding to at least one audio sampling point in the first audio signal.
[0059] Among them, the timestamp corresponding to at least one audio sampling point can be pre-set in a certain manner. For example, the timestamp can be set for the audio sampling points in the first audio signal at a certain time interval (or sampling point interval). The audio sampling point is a sampling point obtained by sampling the analog signal at a certain audio sampling rate. The audio sampling rate refers to the number of times the recording device samples the analog signal per unit time during the audio recording stage. For example, the audio sampling rate is 48000 Hz (Hertz), which means 48000 audio sampling points per second, that is, 48 audio sampling points per millisecond. When timestamping the audio signal, the timestamp can be stamped at the corresponding audio sampling point at a specified time interval, indicating the relative time point of the audio sampling point in the audio signal relative to the starting audio sampling point. The time interval can be set according to actual needs, for example, the time interval can be 1 millisecond, 2 milliseconds, 3 milliseconds, ..., 1 second, 2 seconds, 3 seconds, etc. The specific time interval is not limited. For example, a timestamp is stamped for every 1 second of audio sampling points. In practical applications, a corresponding timestamp can also be set for each audio sampling point.
[0060] The step 2210 of modulating and generating ultrasonic information according to the time information corresponding to the first audio signal may include: modulating and generating ultrasonic information corresponding to at least one audio sampling point according to a timestamp corresponding to at least one audio sampling point in the first audio signal.
[0061] The timestamp of each audio sampling point in at least one audio sampling point with a timestamp may be encoded to obtain an encoded timestamp, for example, the timestamp may be encoded in binary or other digital encoding forms to obtain an encoded timestamp, and the encoded timestamp may be modulated to generate ultrasonic information.
[0062] In this embodiment, the ultrasonic information corresponding to the audio sampling point is generated by modulating the timestamp corresponding to at least one audio sampling point in the first audio signal, so that when the second audio signal is played, when the audio sampling point with ultrasonic information is played, the timestamp of the audio sampling point can be transmitted to the receiving device through ultrasonic waves, providing the receiving device with real-time and accurate current playback time. By setting the timestamps of multiple audio sampling points, the receiving device can be synchronized multiple times at intervals during the video playback process, avoiding or reducing the cumulative errors caused by the heterogeneity of the receiving device and the sending device, which helps to improve the overall synchronization of the video playback and sound playback of the receiving device.
[0063] In some optional embodiments, mixing the first audio signal and the ultrasonic information to generate the second audio signal in step 2220 may include: superimposing the ultrasonic information corresponding to at least one audio sampling point in the first audio signal with the first audio signal to obtain the second audio signal.
[0064] Among them, the ultrasonic information corresponding to each audio sampling point of at least one audio sampling point can be superimposed on the audio signal of the audio sampling point, and the ultrasonic information is added to the audio signal of the corresponding audio sampling point to obtain a second audio signal including the ultrasonic information of at least one audio sampling point.
[0065] In this embodiment, the second audio signal is obtained by mixing the ultrasonic information corresponding to at least one audio sampling point in the first audio signal with the first audio signal, so that the timestamp of each audio sampling point can be transmitted to the receiving device through ultrasonic wave, thereby improving the accuracy of the current playback time based on the video synchronization of the receiving device, and further improving the synchronization of the video playback progress of the receiving device with the sound playback progress of the sending device.
[0066] Figure 4 It is a flowchart of a method for synchronizing audio and video provided by yet another exemplary embodiment of the present disclosure.
[0067] In some optional embodiments, such as Figure 4 As shown, the method of the embodiment of the present disclosure may also include:
[0068] Step 310: Determine the video corresponding to the first audio signal.
[0069] The video corresponding to the first audio signal may include one or more video frames. During video playback, each video frame in the video may be displayed at a certain frame rate to achieve video playback.
[0070] In some optional embodiments, the video corresponding to the first audio signal can be obtained synchronously with the first audio signal. For example, the video and the corresponding first audio signal can be stored together, and the first audio signal and the video corresponding to the first audio signal are obtained according to the storage addresses corresponding to the video and the first audio signal respectively. Alternatively, the video can be obtained separately from the first audio signal. For example, the video and the corresponding first audio signal can be stored separately, and the corresponding relationship between the video and the first audio signal is maintained, and the video corresponding to the first audio signal is determined according to the corresponding relationship.
[0071] In some optional embodiments, a video may be determined first, and then a corresponding first audio signal may be determined based on the video.
[0072] Step 320: Play the corresponding video synchronously with the second audio signal.
[0073] The sending end device can read the video from the video file storing the video, and then play the video synchronously with the second audio signal. On the sending end device, since the playback of the video and the audio are both controlled by the sending end device, the synchronous playback of the video and the audio on the same device can be achieved by any practicable synchronization method in the relevant technology, which is not limited in the embodiments of the present disclosure.
[0074] In this embodiment, the sending device can play the corresponding video synchronously with the second audio signal, thereby achieving synchronization of the audio and video of the sending device. On the basis of ensuring the synchronous audio-visual experience of the second user corresponding to the receiving device, the synchronous audio-visual experience of the first user corresponding to the sending device can also be ensured.
[0075] In some optional embodiments, playing the sound signal corresponding to the second audio signal in step 230 includes: transmitting the second audio signal to a sound player, so that the sound player converts the second audio signal into a sound signal for playing.
[0076] Among them, the sound player can be a sound player on the sending device, or the sound player can be a sound player independent of the sending device and connected to the sending device. The connection method between the sound player and the sending device includes but is not limited to a network connection. The network connection method may include, for example, wireless, wired, etc., which are not specifically limited. The sound player can convert the second audio signal from an electrical signal into a sound signal, so that the sound signal propagates in the air and is transmitted to the receiving device, so that the receiving device can obtain the current playback time from the ultrasonic information of the sound signal. The specific working principle of the sound player is not repeated here.
[0077] In this embodiment, by transmitting the second audio signal to a sound player, the sound signal corresponding to the second audio signal can be played through the sound player, and the timestamp of the current playback progress of the sound can be transmitted to the receiving device through ultrasound, thereby providing the receiving device with real-time and accurate current playback time.
[0078] In some optional embodiments, the ultrasonic information may also include link information of the video corresponding to the first audio signal, so that the receiving device obtains the current playback time and link information from the ultrasonic information, obtains the video based on the link information, and plays the video synchronously with the sending device according to the current playback time.
[0079] The link information of the video may include the storage path of the video file corresponding to the video. The link information of the video corresponding to the first audio signal may be modulated with the time information to generate ultrasonic information, which is added to the first audio signal to obtain the second audio signal, so that the second audio signal includes ultrasonic information, and the ultrasonic information includes the time information corresponding to the first audio signal and the link information of the video. The modulation principle of the link information of the video is similar to that of the time information. The link information of the video may be encoded into a form that can be represented by ultrasonic waves, such as binary or other digital encoding forms, to obtain the encoded link information corresponding to the link information, and the ultrasonic information of the encoded link information may be obtained by modulating the frequency, phase (or amplitude), etc. of the ultrasonic signal, and the ultrasonic information of the time information and the ultrasonic information of the link information may be added to the first audio signal to obtain the second audio signal including the ultrasonic information. The receiving device may obtain the current playback time and link information from the ultrasonic information, and then obtain the video according to the link information, and play the video synchronously with the transmitting device according to the current playback time.
[0080] In some optional embodiments, when the receiving device does not have a video corresponding to the first audio signal locally, the sending device transmits the link information of the video together with the time information to the receiving device through ultrasound, so that the receiving device can obtain the corresponding video according to the link information and play the video synchronously with the sending device.
[0081] In some optional embodiments, referring to the above embodiments, the time information corresponding to the first audio signal may include a timestamp corresponding to at least one audio sampling point. The link information of the video corresponding to the first audio signal may include link information of the video segment corresponding to each timestamp. The video segment corresponding to each timestamp may include a plurality of video frames starting from a video frame (e.g., C11) corresponding to the timestamp (e.g., t1) to a previous frame (e.g., C1n) of a video frame (C21) corresponding to a next timestamp (e.g., t2), where n represents the number of video frames between two adjacent timestamps.
[0082] In this embodiment, the time information and the video link information corresponding to the first audio signal are adjusted to generate ultrasonic information, and the time information and the video link information are transmitted to the receiving device through ultrasonic waves, so that the receiving device can still play the video synchronously with the transmitting device when the video does not exist locally, further improving the experience of the second user corresponding to the receiving device. For example, in the vehicle space, the front-row car machine can obtain the video that the front-row user (first user) wants to watch from the video server through the network, and the rear-row car machine or the rear-row video player does not have the video locally. The current play time and the video link information can be obtained through the sound signal played by the front-row car machine, and the video can be played synchronously with the front-row video player in the rear-row video player.
[0083] In some optional embodiments, the ultrasonic information may also include other relevant information in addition to the time information corresponding to the first audio signal and the link information of the video. For example, the ultrasonic information may also include the operation information of the first user corresponding to the transmitting device on the played video, such as the bullet screen information sent by the first user to the played video, so that the receiving device can play the video and the bullet screen information of the first user synchronously with the transmitting device.
[0084] The above-mentioned embodiments of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.
[0085] Any audio and video synchronization method provided in the embodiments of the present disclosure can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: electronic devices such as terminal devices and servers. Alternatively, any audio and video synchronization method provided in the embodiments of the present disclosure can be executed by a processor, such as the processor executing any audio and video synchronization method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.
[0086] Figure 5 FIG. 1 is a flowchart of a method for synchronizing audio and video provided by another exemplary embodiment of the present disclosure. This embodiment can be applied to an electronic device, such as a video playback device independent of a vehicle computer, and the electronic device serves as a receiving device. Figure 5 As shown, the method of the embodiment of the present disclosure may include the following steps:
[0087] Step 410: collect the sound signal played by the transmitting device to obtain an audio signal corresponding to the sound signal.
[0088] The sound signal played by the transmitting device is the sound signal corresponding to the second audio signal played by the transmitting device in the above embodiment. The sound signal collected at the current time point may be the audio signal of one or more audio sampling points currently played in the second audio signal. That is, the audio signal corresponding to the sound signal collected at a time point may be the audio signal of some audio sampling points of the second audio signal.
[0089] In some optional embodiments, the sound signal played by the transmitting device can be collected by an audio collection device connected to the receiving device, and the sound signal can be converted into a corresponding audio signal. The audio signal can be the same as or similar to the audio signal currently played in the second audio signal of the transmitting device. It should be noted that the audio collection device is a device that can collect sound signals and ultrasonic signals.
[0090] Step 420 : In response to the audio signal including ultrasonic information, determining a current play time corresponding to the sound signal from the ultrasonic information of the audio signal.
[0091] Among them, since the time information corresponding to the first audio signal includes a timestamp corresponding to at least one audio sampling point, if not every audio sampling point has a timestamp, the audio signal collected by the receiving end device may not include ultrasonic information. In an audio signal that does not include ultrasonic information, the receiving end device cannot obtain the current playback time, and therefore, cannot synchronize the video and audio. The receiving end device can continue to play at its current playback speed until the collected audio signal includes ultrasonic information, and the receiving end device determines the current playback time from the ultrasonic information of the audio signal.
[0092] In some optional embodiments, if this is the first time that the receiving device is collecting sound signals and the collected audio signal does not include ultrasonic information, the receiving device can play the video at a preset playback speed until the collected audio signal includes ultrasonic information, and the receiving device determines the current playback time from the ultrasonic information of the audio signal.
[0093] Step 430, synchronously play the corresponding video according to the current play time.
[0094] If the audio signal includes ultrasonic information, the receiving device obtains the current playback time from the ultrasonic information of the audio signal, and can play the corresponding video synchronously according to the current playback time. The specific synchronization method of the receiving device playing the corresponding video synchronously with the sound can refer to the above embodiment, which is not repeated here.
[0095] The audio and video synchronization method provided in this embodiment, since the audio signal includes ultrasonic information, and the ultrasonic information includes the current playback time, the receiving device can obtain the current playback time of the real-time played sound signal from the ultrasonic information of the collected sound signal, and then can play the corresponding video synchronously according to the current playback time, so as to achieve the synchronization of the video played by the receiving device and the sound played by the sending device. And the real-time playback time of the sound signal obtained by the receiving device through ultrasonic transmission can effectively reduce the estimation error and network transmission delay error compared with the estimation time through network transmission in the related art, so as to greatly improve the synchronization of the video playback progress of the receiving device and the sound playback progress of the sending device.
[0096] In some optional embodiments, determining the current playback time from the ultrasonic information of the audio signal in step 420 may include: obtaining ultrasonic information from the audio signal by parsing according to a parsing method corresponding to the audio signal; and determining a timestamp from the ultrasonic information as the current playback time according to a demodulation method corresponding to the ultrasonic information.
[0097] Among them, the parsing method is the inverse process of the mixing process of the first audio signal and the ultrasonic information in the aforementioned embodiment. That is, the parsing method is to parse the ultrasonic frequency band from the audio signal and obtain the ultrasonic information from the ultrasonic frequency band. The demodulation method is the inverse process of the modulation process of modulating the time information to generate the ultrasonic information in the aforementioned embodiment, that is, demodulating the encoded timestamp corresponding to the audio sampling point from the ultrasonic information, and decoding according to the encoded timestamp to obtain the original timestamp. For example, if the timestamp is 300 milliseconds after the 5th second, it means that the current playback time of the sound is 300 milliseconds after the 5th second, and the receiving end device can update the video played by the receiving end device to the position of 300 milliseconds after the 5th second to ensure the synchronization of the video and audio of the receiving end device.
[0098] In some optional embodiments, the ultrasonic information may also include link information of the video corresponding to the sound signal.
[0099] Determining the timestamp from the ultrasonic information according to a demodulation method corresponding to the ultrasonic information may include: determining the timestamp and video link information from the ultrasonic information according to the demodulation method.
[0100] Synchronously playing the corresponding video according to the current playing time may include: acquiring the video corresponding to the sound signal according to the link information, and synchronously playing the video corresponding to the sound signal according to the current playing time.
[0101] Among them, the demodulation method is the reverse process of modulating the timestamp and the link information of the video to generate ultrasonic information. The details are not described in detail. The link information of the video obtained by demodulation can be found in the aforementioned embodiment and will not be described here. The timestamp obtained by demodulation is determined as the current playback time of the sound signal. The receiving device obtains the video corresponding to the sound signal according to the link information, and plays the video corresponding to the sound signal synchronously according to the current playback time. In the case that the video corresponding to the first audio signal does not exist locally on the receiving device, the transmitting device transmits the link information of the video to the receiving device through ultrasound, so that the receiving device can obtain the video corresponding to the sound signal according to the link information, and play the video synchronously with the transmitting device.
[0102] In some optional embodiments, Figure 6 FIG. 1 is a flowchart of a method for synchronizing audio and video provided by an exemplary embodiment of the present disclosure. Figure 6 As shown, the process of the method may include the following steps:
[0103] Step 510: The transmitting end device determines a first audio signal and a first video to be played.
[0104] Step 520: The transmitting end device processes the first audio signal to generate a second audio signal including ultrasonic information.
[0105] The ultrasonic information includes time information corresponding to the first audio signal.
[0106] Step 530: The transmitting end device plays a sound signal corresponding to the second audio signal, and plays a corresponding video synchronously with the second audio signal.
[0107] Step 540: The receiving device collects a sound signal corresponding to the second audio signal played by the transmitting device to obtain an audio signal corresponding to the sound signal.
[0108] Step 550: In response to the audio signal including ultrasonic information, the receiving end device determines the current playing time corresponding to the sound signal from the ultrasonic information of the audio signal.
[0109] Step 560: The receiving device synchronously plays the corresponding video according to the current playing time.
[0110] The specific operations of the above steps 510 to 560 can be found in the above embodiments and will not be described in detail here.
[0111] In some optional embodiments, in the case of multiple receiving devices, each receiving device can obtain the current playback time from the ultrasonic information of the sound signal by collecting the sound signal played by the transmitting device according to the method of the embodiment of the present disclosure, and realize the synchronous playback of the video and sound of the receiving device according to the current playback time. Since multiple receiving devices can play videos synchronously with the sound of the transmitting device, and the transmitting device can also play videos synchronously with the sound, it is possible to realize multi-screen interaction in any multimedia scene, so that multiple users in the scene can watch the same video synchronously on different video playback devices while sharing a set of sound playback devices. In addition, the solution of realizing multi-screen interaction through network transmission in the related art depends on the development of both ends, that is, the transmitting device and the receiving device need to perform network communication and corresponding processing, and the development is relatively complex. The embodiment of the present disclosure can avoid the network communication between the transmitting device and the receiving device, but through the ultrasonic transmission timestamp and video link information carried by the sound signal, the development complexity of the devices at both ends is effectively reduced.
[0112] The above-mentioned embodiments of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.
[0113] Any audio and video synchronization method provided in the embodiments of the present disclosure can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: electronic devices such as terminal devices and servers. Alternatively, any audio and video synchronization method provided in the embodiments of the present disclosure can be executed by a processor, such as the processor executing any audio and video synchronization method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.
[0114] Exemplary Devices
[0115] Figure 7 1 is a schematic diagram of the structure of an audio and video synchronization device provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure, and the device of this embodiment can be set in a transmitting end device. Figure 7 The device shown may include: a first determination module 610 , a first processing module 620 and a play control module 630 .
[0116] The first determination module 610 may be configured to determine a first audio signal to be played.
[0117] In some optional embodiments, the first determination module 610 may be a module in a signal processor. For example, the first determination module 610 may be a data interface of the signal processor, which may read the first audio signal to be played from a storage area, or receive the first audio signal to be played from other devices or components.
[0118] The first processing module 620 may be configured to process the first audio signal to generate a second audio signal including ultrasonic information; the ultrasonic information includes time information corresponding to the first audio signal.
[0119] In some optional embodiments, the first processing module 620 may be a module in a signal processor having a corresponding signal processing function.
[0120] The play control module 630 can be used to play the sound signal corresponding to the second audio signal, so that the receiving device determines the current play time from the ultrasonic information of the sound signal and synchronously plays the corresponding video according to the current play time.
[0121] In some optional embodiments, the playback control module 630 may be any controller, microcontroller, etc. that can implement audio playback control.
[0122] Figure 8 It is a structural diagram of an audio and video synchronization device provided by another exemplary embodiment of the present disclosure.
[0123] In some optional embodiments, in the above Figure 7 Based on the embodiment shown, Figure 8 As shown, the first processing module 620 may include: a modulation unit 6210 and a mixing unit 6220 .
[0124] The modulation unit 6210 may be configured to generate ultrasonic information by modulation according to time information corresponding to the first audio signal.
[0125] In some optional embodiments, the modulation unit 6210 may be a unit in a signal processor having an ultrasonic modulation function.
[0126] The audio mixing unit 6220 may be configured to generate a second audio signal by mixing the first audio signal and the ultrasonic information.
[0127] In some optional embodiments, the mixing unit 6220 may be a unit in a signal processor having a corresponding mixing function.
[0128] In some optional embodiments, the time information corresponding to the first audio signal may include a timestamp corresponding to at least one audio sampling point in the first audio signal.
[0129] The modulation unit 6210 may be specifically configured to: modulate and generate ultrasonic wave information corresponding to at least one audio sampling point according to a timestamp corresponding to at least one audio sampling point in the first audio signal.
[0130] In some optional embodiments, the audio mixing unit 6220 may be specifically configured to: superimpose ultrasonic information corresponding to at least one audio sampling point in the first audio signal with the first audio signal to obtain a second audio signal.
[0131] Fig. 9 4 is a schematic diagram of the structure of an audio and video synchronization device provided by another exemplary embodiment of the present disclosure.
[0132] In some optional embodiments, such as Fig. 9 As shown, the apparatus of the embodiment of the present disclosure may further include: a second determination module 710.
[0133] The second determination module 710 is configured to determine the video corresponding to the first audio signal.
[0134] In some optional embodiments, the second determination module 710 and the first determination module 610 may be the same module or different modules.
[0135] The play control module 630 may also be used to play the corresponding video synchronously with the second audio signal.
[0136] In some optional embodiments, the playback control module 630 may be any controller, microcontroller, etc. having a video playback control function.
[0137] In some optional embodiments, the play control module 630 may be specifically configured to: transmit the second audio signal to a sound player, so that the sound player converts the second audio signal into a sound signal for play.
[0138] In some optional embodiments, the ultrasonic information may also include link information of the video corresponding to the first audio signal, so that the receiving device obtains the current playback time and the link information of the video from the ultrasonic information, obtains the video according to the link information, and plays the video synchronously with the sending device according to the current playback time.
[0139] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.
[0140] Fig.10 FIG. 1 is a schematic diagram of a structure of an audio and video synchronization device provided by another exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure, and the device of this embodiment can be set in a receiving end device. Fig.10 The device shown may include: a signal acquisition module 810 , a second processing module 820 and a synchronization control module 830 .
[0141] The signal acquisition module 810 may be used to acquire the sound signal played by the transmitting device and obtain the audio signal corresponding to the sound signal.
[0142] In some optional embodiments, the signal acquisition module 810 may be a specific audio acquisition device capable of acquiring sound signals and ultrasonic signals. For example, the signal acquisition module 810 may be a microphone, a microphone array, etc. The frequency of ultrasonic waves is higher than the frequency range of sound that can be perceived by human hearing. Therefore, the frequency response range of the audio acquisition device needs to include ultrasonic waves so that ultrasonic signals can be acquired, thereby obtaining ultrasonic information in the audio signal.
[0143] The second processing module 820 may be configured to determine a current playing time corresponding to the sound signal from the ultrasonic information of the audio signal in response to the audio signal including ultrasonic information.
[0144] In some optional embodiments, the second processing module 820 may be a module in a signal processor having corresponding signal analysis and demodulation functions.
[0145] The synchronization control module 830 can be used to synchronously play the corresponding video according to the current playing time.
[0146] In some optional embodiments, the synchronization control module 830 may be any controller, microcontroller, etc. that can implement the video playback control function.
[0147] In some optional embodiments, the second processing module 820 can be specifically used to: parse the audio signal to obtain ultrasonic information according to a parsing method corresponding to the audio signal; and determine a timestamp from the ultrasonic information as the current playback time according to a demodulation method corresponding to the ultrasonic information.
[0148] In some optional embodiments, the ultrasonic information may also include link information of the video corresponding to the sound signal.
[0149] The second processing module 820 can be specifically used to determine the timestamp and video link information from the ultrasonic information according to the demodulation method.
[0150] The synchronization control module 830 can be specifically used to: obtain the video corresponding to the sound signal according to the link information, and synchronously play the video corresponding to the sound signal according to the current playing time.
[0151] In some optional embodiments, Fig.11 FIG. 1 is a schematic diagram of a structure of an audio and video synchronization system provided by an exemplary embodiment of the present disclosure. Fig.11As shown, the system may include an audio and video synchronization device 70, a sound player 71 and a video player 72 connected to the audio and video synchronization device 70, an audio and video synchronization device 80, and an audio acquisition device 81 and a video player 82 connected to the audio and video synchronization device 80. Among them, the audio and video synchronization device 70 is used for a sending end device, where the sending end device can be a device independent of the video player 72. For example, the sending end device can be a chip for signal processing and control in the vehicle computer to realize the control of the sound player 71 and the video player 72. The audio and video synchronization device 80 is used for a receiving end device. Here, the receiving end device can be a device independent of the video player 82. For example, the receiving end device can be a controller or a chip connected to the video player 82. When user A plays a video through the video player 72, the audio and video synchronization device 70 can determine the first audio signal to be played and the corresponding audio signal according to the selection operation of user A. The audio and video synchronization device 70 processes the first audio signal to generate a second audio signal including ultrasonic information. The audio and video synchronization device 70 sends the second audio signal to the sound player 71, and sends the video to the video player 72, controls the sound player 71 to play the sound signal corresponding to the second audio signal, and controls the video player 72 to synchronously play the corresponding video. When user B watches the video through the video player 82, the audio and video synchronization device 80 collects the sound signal played by the sound player 71 through the audio acquisition device 81 to obtain the audio signal corresponding to the sound signal. The audio and video synchronization device 80 analyzes and processes the audio signal. If it is determined that the currently collected audio signal includes ultrasonic information, the current playback time corresponding to the sound signal is determined from the ultrasonic information of the audio signal, and according to the current playback time, the video player 82 is controlled to synchronously play the corresponding video. If the currently collected audio signal does not include ultrasonic information, the video player 82 is controlled to continue playing according to the current playback progress until the collected audio signal includes ultrasonic information. According to the current playback time in the ultrasonic information, the video playback speed of the video player 82 is controlled. For example, if the current video playback progress of the video player 82 is slower than the sound playback progress, the video playback speed of the video player 82 can be increased. If the current video playback progress is ahead of the sound playback progress, the video playback speed of the video player 82 can be reduced, thereby continuously reducing the time difference between the playback progress of the video player 82 and the playback progress of the sound signal, and achieving the effect of synchronous playback. Alternatively, the video player 82 can be directly controlled to update the playback progress to the video progress corresponding to the current playback time according to the current playback time. In the case of determining the playback progress of the sound, the specific synchronization process of the video playback progress and the sound playback progress of the receiving device is not limited.During the entire video playback process, the sound signal can transmit the playback time once every period of time through ultrasound. The audio and video synchronization device 80 can perform synchronization every time the current playback time is obtained, thereby ensuring the overall synchronization of the video playback progress of the video player 82 and the sound playback progress of the sound player 71.
[0152] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.
[0153] Exemplary Electronic Devices
[0154] Fig.12 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure, including at least one processor 91 and a memory 92.
[0155] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.
[0156] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.
[0157] In one example, the electronic device 90 may further include: an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0158] The input device 93 may also include, for example, a touch screen, a microphone, various sensors, and the like. Sensors may include, for example, image sensors (such as cameras, cameras, etc.), laser radars, millimeter-wave radars, ultrasonic radars, positioning sensors, pressure sensors, air quality sensors, temperature sensors, and the like. Image sensors, laser radars, millimeter-wave radars, ultrasonic radars, and the like may be used to sense the surrounding environment, that is, to detect dynamic and static objects in the surrounding environment. Dynamic and static objects may include, for example, static objects such as lane lines, curbs, arrows, signs, trees, and buildings, as well as dynamic objects such as surrounding vehicles, pedestrians, and cyclists. Positioning sensors are used to realize the positioning of movable devices (such as self-vehicles, robots, etc.) where electronic devices are located. Positioning sensors may include, for example, inertial measurement units (Inertial Measurement Unit, referred to as: IMU), global positioning systems (Global Positioning System, referred to as: GPS), and the like. Pressure sensors may be used to detect seat pressure. Temperature sensors may be used to detect the temperature in the vehicle cabin. Air quality sensors may be used to detect the air quality in the vehicle cabin.
[0159] The output device 94 can output various information to the outside, and may include, for example, a display, a speaker, a communication network and a remote output device connected thereto, and the like.
[0160] Of course, to simplify, Fig.12 Only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.
[0161] Exemplary computer program products and computer-readable storage media
[0162] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the methods of various embodiments of the present disclosure described in the above “Exemplary Methods” section.
[0163] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0164] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.
[0165] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0166] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.
[0167] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A method for synchronizing audio and video, for use in a transmitting device, the method comprising: Determining a first audio signal to be played; Processing the first audio signal to generate a second audio signal including ultrasonic information; The ultrasonic information includes time information corresponding to the first audio signal; The sound signal corresponding to the second audio signal is played, so that the receiving end device determines the current playing time from the ultrasonic information of the sound signal, and synchronously plays the corresponding video according to the current playing time.
2. The method according to claim 1, wherein: The processing of the first audio signal to generate a second audio signal including ultrasonic information includes: modulating and generating ultrasonic information according to the time information corresponding to the first audio signal; The second audio signal is generated by mixing the first audio signal and the ultrasonic information.
3. The method according to claim 2, wherein: The time information corresponding to the first audio signal includes a timestamp corresponding to at least one audio sampling point in the first audio signal; The step of modulating and generating ultrasonic information according to the time information corresponding to the first audio signal includes: According to the timestamp corresponding to the at least one audio sampling point in the first audio signal, the ultrasonic information corresponding to the at least one audio sampling point is modulated and generated.
4. The method according to claim 3, wherein: The step of mixing the first audio signal and the ultrasonic information to generate the second audio signal includes: The ultrasonic information corresponding to the at least one audio sampling point in the first audio signal is superimposed on the first audio signal to obtain the second audio signal.
5. The method according to any one of claims 1 to 4, wherein: Also includes: Determining a video corresponding to the first audio signal; The corresponding video is played synchronously with the second audio signal.
6. The method according to any one of claims 1 to 4, wherein: The ultrasonic information also includes link information of the video corresponding to the first audio signal, so that the receiving device obtains the current playback time and the link information from the ultrasonic information, obtains the video according to the link information, and plays the video synchronously with the sending device according to the current playback time.
7. A method for synchronizing audio and video, for use in a receiving device, the method comprising: Collecting a sound signal played by a transmitting device to obtain an audio signal corresponding to the sound signal; In response to the audio signal including ultrasonic information, determining a current play time corresponding to the sound signal from the ultrasonic information of the audio signal; The corresponding video is played synchronously according to the current playing time.
8. An audio and video synchronization device, used in a transmitting end device, the device comprising: A first determining module, used to determine a first audio signal to be played; A first processing module, configured to process the first audio signal to generate a second audio signal including ultrasonic information; the ultrasonic information includes time information corresponding to the first audio signal; The play control module is used to play the sound signal corresponding to the second audio signal, so that the receiving end device determines the current play time from the ultrasonic information of the sound signal, and synchronously plays the corresponding video according to the current play time.
9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 7.
10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 7.
