Sound and picture synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to a sound and picture synchronization method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining first video data of camera equipment and second audio data of pickup equipment; video time stamps of the video equipment are added to the first video data and the second audio data; analyzing the first video data based on the video timestamp to obtain a first time interval corresponding to each speech; analyzing the second audio data based on the video timestamp to obtain a second time interval corresponding to each speech; and according to the first time interval and the second time interval, synchronizing the second audio data into the first video data to obtain target audio and video data. According to the audio and picture synchronization method and device, the problem that the video picture jumps or is stuck due to the fact that the video frame rate is unstable is solved, the reliability of audio and picture synchronization is improved, and the presentation effect of the synchronized video picture is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to audio-visual synchronization methods, apparatus, computer devices, and storage media. Background Technology

[0002] In applications such as video conferencing and classrooms, the requirements for audio fidelity are gradually increasing, making the audio data recorded by the microphones built into network cameras (IPCs) insufficient. This necessitates the use of professional audio pickup equipment to acquire audio data. Since the video data of the IPC and the audio data of the pickup equipment are acquired through different devices, if the pickup equipment experiences stuttering or other issues during the acquisition of audio data, the final synthesized video may have problems with audio and video synchronization.

[0003] The current solution for achieving audio-visual synchronization is to dynamically adjust the rendering timing of video frames based on the audio playback time of the audio pickup device. Audio is continuously output through the player, and the timestamps of the video frames must match the current audio rendering progress. If a video frame is too early, it is waited for; if it is too late, the frame is discarded. The drawback of this solution is that if the video frame rate is unstable (e.g., network fluctuations causing delayed video frames), frames with excessive delays must be discarded, resulting in skipping or stuttering in the video.

[0004] There is currently no effective solution to the problem of unstable video frame rates in related technologies, which causes video footage to jump or stutter. Summary of the Invention

[0005] This embodiment provides an audio-visual synchronization method, apparatus, computer device, and storage medium to solve the problem of unstable video frame rate in related technologies, which causes video images to jump or stutter.

[0006] In a first aspect, this embodiment provides an audio-visual synchronization method applied to an audio-visual synchronization system; the audio-visual synchronization system includes a recording device, a sound pickup device connected to the recording device, and a camera device; the method is applicable to the recording device, and the method includes:

[0007] The system acquires first video data from the camera device and second audio data from the audio pickup device; the first video data and the second audio data are appended with the recording timestamp of the recording device.

[0008] Based on the recording timestamp, the first video data is analyzed to obtain the first time interval corresponding to each speech;

[0009] Based on the video timestamp, the second audio data is analyzed to obtain the second time interval corresponding to each speech;

[0010] Based on the first time interval and the second time interval, the second audio data is synchronized to the first video data to obtain the target audio and video data.

[0011] In some embodiments, acquiring first video data from the camera device and second audio data from the microphone device includes:

[0012] Acquire the audio and video data of the camera device and the second audio data of the sound pickup device, and add the recording timestamp of the recording device to the audio and video data and the second audio data;

[0013] The audio and video data are stripped to obtain first video data and first audio data.

[0014] In some embodiments, the method further includes:

[0015] After obtaining the first video data and the first audio data, it is determined whether the first condition and the second condition are met; the first condition is whether the recording device is acquiring the first video data and the second audio data for the first time; the second condition is whether the sound pickup device or the camera device has disconnected and reconnected.

[0016] If at least one condition is met, the first video data, the first audio data, and the second audio data are cached in the cache area.

[0017] If neither of the two conditions is met, then the first video data and the second audio data are analyzed in real time.

[0018] In some embodiments, the first video data is analyzed based on the recording timestamp to obtain a first time interval corresponding to each speech, including:

[0019] Using the pauses in the target's speech in the first video data as anchor points, and based on the recording timestamps, speech analysis is performed on the first video data to obtain the first timestamp of each speech.

[0020] Record the time interval between adjacent first timestamps to obtain the first time interval corresponding to each speech.

[0021] In some embodiments, the second audio data is analyzed based on the recording timestamp to obtain a second time interval corresponding to each speech, including:

[0022] Using the pauses in the target's speech in the second audio data as anchor points, and based on the video recording timestamps, speech analysis is performed on the second audio data to obtain the second timestamp for each speech;

[0023] Record the time interval between adjacent second timestamps to obtain the second time interval corresponding to each speech.

[0024] In some embodiments, the second audio data is synchronized to the first video data according to the first time interval and the second time interval to obtain target audio and video data, including:

[0025] Based on the first time interval and the second time interval, determine whether the target objects in the first video data and the second audio data correspond;

[0026] If the target object in the first video data and the second audio data does not correspond, the first video data and the acquired first audio data are encoded and stored.

[0027] If the target objects in the first video data and the second audio data correspond, the initial speech time difference is recorded and used as the synchronization time difference; the initial speech time difference is the time difference of the first speech of the target object in the first video data and the second audio data.

[0028] Keeping the recording timestamp of the first video data unchanged, the recording timestamp of the second audio data is superimposed with the synchronization time difference to obtain the target audio and video data.

[0029] In some embodiments, the method further includes:

[0030] When recording the initial speech time difference, the end speech time difference is also recorded; the end speech time difference is the time difference of the last speech by the target object in the first video data and the second audio data;

[0031] The synchronization time difference is adjusted based on the initial speech time difference and the end speech time difference.

[0032] Secondly, this embodiment provides an audio-visual synchronization device for use in an audio-visual synchronization system; the audio-visual synchronization system includes a recording device, a sound pickup device connected to the recording device, and a camera device; the device is applicable to the recording device and includes: an acquisition module, a first analysis module, a second analysis module, and a synchronization module;

[0033] The acquisition module is used to acquire first video data from the camera device and second audio data from the audio pickup device; the first video data and the second audio data are appended with the recording timestamp of the recording device;

[0034] The first analysis module is used to analyze the first video data based on the recording timestamp to obtain the first time interval corresponding to each speech;

[0035] The second analysis module is used to analyze the second audio data based on the video recording timestamp to obtain the second time interval corresponding to each speech;

[0036] The synchronization module is used to synchronize the second audio data to the first video data according to the first time interval and the second time interval to obtain the target audio and video data.

[0037] Thirdly, this embodiment provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the audio-visual synchronization method described in the first aspect above.

[0038] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the audio-visual synchronization method described in the first aspect above.

[0039] Compared with related technologies, the audio-visual synchronization method, apparatus, computer equipment, and storage medium provided in this embodiment are applied to an audio-visual synchronization system. The audio-visual synchronization system includes a recording device, a sound pickup device connected to the recording device, and a camera device. The method is applicable to the recording device. The method involves: acquiring first video data from the camera device and second audio data from the sound pickup device; adding a recording timestamp from the recording device to the first video data and the second audio data; analyzing the first video data based on the recording timestamp to obtain a first time interval corresponding to each speech; analyzing the second audio data based on the recording timestamp to obtain a second time interval corresponding to each speech; and synchronizing the second audio data to the first video data according to the first and second time intervals to obtain target audio-visual data. This solves the problem of unstable video frame rates, leading to video skipping or stuttering. By using the recording timestamp as the synchronization time basis, the reliability of audio-visual synchronization is improved. Furthermore, by using the first video data as the video basis for synchronization based on the first and second time intervals, audio-visual synchronization is completed, improving the presentation effect of the synchronized video.

[0040] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0041] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0042] Figure 1 This is a hardware structure block diagram of a terminal device for an audio-visual synchronization method provided in an embodiment of this application;

[0043] Figure 2 This is a flowchart of an audio-visual synchronization method provided in an embodiment of this application;

[0044] Figure 3 This is a flowchart of step S210;

[0045] Figure 4 This is a flowchart of step S220;

[0046] Figure 5 This is a flowchart of step S230;

[0047] Figure 6 This is a flowchart of a preferred embodiment of the audio-visual synchronization method provided in this application;

[0048] Figure 7 It is a comparison chart of video frames and audio frames;

[0049] Figure 8 This is a structural block diagram of an audio-visual synchronization device provided in an embodiment of this application.

[0050] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 210, acquisition module; 220, first analysis module; 230, second analysis module; 240, synchronization module. Detailed Implementation

[0051] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0052] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0053] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the audio-visual synchronization method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0054] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the audio-visual synchronization method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0055] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0056] This embodiment provides a method for audio-visual synchronization. Figure 2 This is a flowchart of the audio-visual synchronization method in this embodiment, as follows: Figure 2 As shown, the process includes the following steps:

[0057] Step S210: Obtain first video data from the camera device and second audio data from the audio pickup device; add a recording timestamp from the recording device to the first video data and the second audio data;

[0058] Step S220: Based on the recording timestamp, analyze the first video data to obtain the first time interval corresponding to each speech;

[0059] Step S230: Based on the video recording timestamp, analyze the second audio data to obtain the second time interval corresponding to each speech;

[0060] Step S240: Synchronize the second audio data into the first video data according to the first time interval and the second time interval to obtain the target audio and video data.

[0061] Specifically, this method embodiment is applied to an audio-visual synchronization system; the audio-visual synchronization system includes a recording device, an audio pickup device connected to the recording device, and a camera device; the method embodiment is applicable to the recording device. The recording device includes, but is not limited to, network hard disk recorders, network cameras, handheld smart imaging devices, etc. The audio pickup device includes, but is not limited to, microphones, etc. The camera device includes, but is not limited to, mobile phones, camcorders, etc. The recording device and the audio pickup device have a one-to-one relationship; while the audio pickup device and the camera device can have a one-to-many relationship. For example, in a video conferencing scenario, there may be multiple camera devices in a conference room, and they can all use the audio from the same microphone.

[0062] The recording device connects to both a camera and an audio pickup device to acquire first video data from the camera and second audio data from the audio pickup device. The first video data and second audio data are appended with the recording timestamp from the recording device, which serves as the basis for subsequent calculations, thus ensuring reliable synchronization of the timestamps in the first video data and second audio data. The first video data can be extracted from the audio and video data of the camera, and the specific processing method is not limited.

[0063] Then, using the recording timestamp as the benchmark for time synchronization, the first video data is analyzed using pre-trained neural network models, video recognition algorithms, and other methods to obtain the first time interval corresponding to each speech. Specifically, the image in each frame of the first video data is analyzed based on whether the target's mouth is moving. The time from stillness to movement is considered one speech time, and each speech is recorded to obtain the corresponding first time interval. The first time interval is the time interval between each speech in the first video data.

[0064] Then, using the video timestamp as the benchmark for time synchronization, the second audio data is analyzed using a pre-trained neural network model or audio recognition algorithm to obtain the second time interval corresponding to each speech. Specifically, each frame of the first audio data is analyzed based on whether there is a target object's voice. The time from the presence of sound to the absence of sound is considered one speech, and each speech is recorded to obtain the corresponding second time interval. The second time interval is the time interval between each speech in the first audio data.

[0065] After obtaining the first and second time intervals, the second audio data can be synchronized to the first video data based on these intervals to obtain the target audio and video data. This completes audio-visual synchronization, improving the presentation of the synchronized video. Since the first video data is used as the synchronization basis, and the recording timestamp is used as the synchronization time basis, the reliability of audio-visual synchronization is improved. This solves the problem of unstable video frame rates causing video skipping or stuttering.

[0066] In related technologies, the solution for achieving audio-visual synchronization involves dynamically adjusting the rendering timing of video frames based on the audio playback time of the audio pickup device. Audio is continuously output through the player, and the timestamps of the video frames must match the current audio rendering progress. If a video frame is too early, it is waited for; if it is too late, the frame is discarded. The drawback of this solution is that if the video frame rate is unstable (e.g., network fluctuations causing delayed video frames), frames with excessive delays must be discarded, resulting in skipping or stuttering in the video. In this embodiment, first video data from a camera device and second audio data from a microphone device are acquired. A recording timestamp is added to both the first and second audio data. Based on the recording timestamp, the first video data is analyzed to obtain a first time interval corresponding to each speech. Similarly, based on the recording timestamp, the second audio data is analyzed to obtain a second time interval corresponding to each speech. The second audio data is synchronized to the first video data according to the first and second time intervals to obtain the target audio-visual data. This solves the problem of unstable video frame rates, which can cause video skipping or stuttering. Using the recording timestamp as the synchronization time basis improves the reliability of audio-visual synchronization. Furthermore, by using the first video data as the video basis for synchronization based on the first and second time intervals, audio-visual synchronization is completed, improving the presentation effect of the synchronized video.

[0067] The steps described above are explained in detail below:

[0068] In some of these embodiments, such as Figure 3 As shown, step S210, acquiring the first video data from the camera device and the second audio data from the microphone device, includes the following steps:

[0069] Step S211: Obtain the audio and video data from the camera device and the second audio data from the sound pickup device, and add the recording timestamp of the recording device to the audio and video data and the second audio data;

[0070] Step S212: The audio and video data are stripped to obtain the first video data and the first audio data.

[0071] Specifically, the second audio data from the sound pickup device can be considered as the audio stream captured by the sound pickup device from the target scene. The first video data from the camera device can be considered as the result of processing the audio and video data captured by the camera device. The processing of audio and video data can be completed in the camera device or in the recording device; there is no restriction on this.

[0072] In this embodiment, to fully utilize the processing performance of the recording device and ensure the reliability of audio-visual synchronization, the audio and video data captured by the camera and the second audio data collected by the microphone are processed within the recording device. Specifically, the process involves acquiring the audio and video data from the camera and the second audio data from the microphone, and simultaneously adding a recording timestamp from the recording device to both the audio and video data and the second audio data, ensuring consistency between their timestamps. Then, using a pre-trained neural network model, an audio-video separation algorithm, and an audio-video processor, the audio and video data are separated to obtain first video data and first audio data. Since the recording timestamp is unique at the current time, there is only one timestamp in both the audio and video data and the second audio data, ensuring reliable synchronization of the timestamps in both the first video data and the second audio data, thus providing a computational foundation for subsequent calculations.

[0073] In some embodiments, the audio-visual synchronization method further includes the following steps:

[0074] After obtaining the first video data and the first audio data, it is determined whether the first condition and the second condition are met; the first condition is whether the recording device is acquiring the first video data and the second audio data for the first time; the second condition is whether the audio pickup device or the camera device has disconnected and reconnected.

[0075] If at least one condition is met, the first video data, the first audio data, and the second audio data are cached in the cache area;

[0076] If neither of the two conditions is met, then the first video data and the second audio data are analyzed in real time.

[0077] Because network environments vary greatly across different usage scenarios, and some network environments may experience network fluctuations, two conditions (a first condition and a second condition) are set to switch the processing strategy for the first video data and the second audio data. The first condition is whether the recording device is acquiring the first video data and second audio data for the first time; this condition can assess whether the recording device has just been powered on. The second condition is whether the audio pickup or camera device has experienced a disconnection and reconnection; if the data collected by the audio pickup or camera device is intermittent, it can be considered that a disconnection and reconnection has occurred. Of course, other methods can also be used to determine this, and there are no restrictions on this.

[0078] If at least one of the first and second conditions is met (including both conditions), the network is considered unstable and cannot meet the requirements for real-time analysis. To ensure audio-visual synchronization, the first video data, first audio data, and second audio data are cached in a buffer area; data is then retrieved from the buffer area for analysis. If neither the first nor the second condition is met, the network is considered stable and meets the requirements for real-time analysis, in which case real-time analysis of the first video data and the second audio data can be performed. Real-time analysis consists of the analysis of the second audio data and the analysis of the first video data, as detailed in steps S220 and S230.

[0079] This embodiment enables flexible switching between real-time analysis and cached analysis, reduces the requirements for network stability, and ensures the stability of system operation.

[0080] In some of these embodiments, such as Figure 4 As shown, step S220, which involves analyzing the first video data based on the recording timestamp to obtain the first time interval corresponding to each speech, includes the following steps:

[0081] Step S221: Using the pauses in the target's speech in the first video data as anchor points and the recording timestamps as a basis, perform speech analysis on the first video data to obtain the first timestamp of each speech.

[0082] Step S222: Record the time interval between adjacent first timestamps to obtain the first time interval corresponding to each speech.

[0083] In this embodiment, a video recognition algorithm can be used. The algorithm analyzes whether the mouth is moving. If the mouth moves from still to moving for more than a first preset value (which can be 100ms, adjustable according to the application scenario), a speech start is recorded, along with the corresponding third timestamp. This yields the corresponding third timestamps t1vb, t2vb, ... for each speech start. If the mouth moves back to still for more than a second preset value (which can be 100ms, adjustable according to the application scenario), a speech end is recorded, along with the corresponding fourth timestamp. This yields the corresponding fourth timestamps t1ve, t2ve, ... (where t represents time, v represents video analysis, b represents start, and e represents end). The interval between the third and fourth timestamps is then recorded to obtain the first time interval for each speech, expressed as: t1ve - t1vb = Δt1v, t2ve - t2vb = Δt2v, ...

[0084] In this embodiment, by using the pauses in the target's speech in the first video data as anchor points and analyzing based on the recording timestamps, the first timestamp of each speech can be quickly and accurately identified.

[0085] In some of these embodiments, such as Figure 5 As shown, step S230, which involves analyzing the second audio data based on the recording timestamp to obtain the second time interval corresponding to each speech, includes the following steps:

[0086] Step S231: Using the pauses in the target's speech in the second audio data as anchor points and the recording timestamp as a basis, perform speech analysis on the second audio data to obtain the second timestamp for each speech.

[0087] Step S232: Record the time interval between adjacent second timestamps to obtain the second time interval corresponding to each speech.

[0088] In this embodiment, an audio recognition algorithm can be used. The algorithm analyzes whether there is a human voice. If a voice appears and disappears within a time frame exceeding a third preset value (which can be 100ms and adjusted according to the application scenario), a speech is recorded as starting, and the corresponding fifth timestamp is recorded, resulting in timestamps t1ab, t2ab, ... for each speech start. If a voice disappears and disappears within a time frame exceeding a fourth preset value (which can be 100ms and adjusted according to the application scenario), a speech is recorded as ending, and the corresponding sixth timestamp is recorded, resulting in timestamps t1ae, t2ae, ... for each speech start (Note: t represents time, a represents audio analysis, b represents start, and e represents end). The interval between the fifth and sixth timestamps is then recorded to obtain the second time interval for each speech, expressed as: t1ae - t1ab = Δt1a, t2ae - t2ab = Δt2a, ...

[0089] In this embodiment, by using the pauses in the target's speech in the second audio data as anchor points and analyzing based on the video recording timestamps, the second timestamp of each speech can be quickly and accurately identified.

[0090] In some embodiments, step S240, which synchronizes the second audio data to the first video data according to a first time interval and a second time interval to obtain the target audio and video data, includes the following steps:

[0091] Step S241: Determine whether the target objects in the first video data and the second audio data correspond based on the first time interval and the second time interval;

[0092] Step S242: If the target object in the first video data and the second audio data does not correspond, then the first video data and the acquired first audio data are encoded and stored.

[0093] Step S243: If the target objects in the first video data and the second audio data correspond, then record the initial speech time difference and use the initial speech time difference as the synchronization time difference; the initial speech time difference is the time difference between the first speech of the target object in the first video data and the second audio data.

[0094] Step S244: Keep the recording timestamp of the first video data unchanged, and add the synchronization time difference to the recording timestamp of the second audio data to obtain the target audio and video data.

[0095] Specifically, to ensure audio-visual synchronization, it's necessary to first determine whether the target objects in the first video data and the second audio data correspond. In this embodiment, if the first time interval and the second time interval are all equal (Δt1v = Δt1a, Δt2v = Δt2a…), it can be assumed that the duration of each speech segment in the first video data is the same as the duration of each speech segment in the second audio data. At this point, it can be concluded that the voice in the second audio data is spoken by the person in the current video frame of the first video data, thus confirming the correspondence between the target objects in the first video data and the second audio data. Furthermore, a certain error is allowed between the first time interval and the second time interval; this error can be set to 25ms.

[0096] If the target object in the first video data and the second audio data does not correspond, the first video data and the acquired first audio data are encoded to obtain archived audio and video data and stored.

[0097] If the target objects in the first video data and the second audio data correspond, the initial speech time difference is recorded and used as the synchronization time difference ΔT. The initial speech time difference is the time difference between the first speech of the target object in the first video data and the second audio data. Keeping the recording timestamp of the first video data unchanged, the recording timestamp of the second audio data is superimposed with the synchronization time difference to obtain the target audio and video data.

[0098] This embodiment first accurately determines whether the target objects in the first video data and the second audio data correspond, and then combines different situations to achieve audio-visual synchronization, reducing the difficulty of synchronization.

[0099] In some embodiments, the audio-visual synchronization method further includes the following steps:

[0100] When recording the initial speech time difference, the end speech time difference is also recorded; the end speech time difference is the time difference between the last speech of the target object in the first video data and the second audio data.

[0101] Adjust the synchronization time difference based on the initial speech time difference and the end speech time difference.

[0102] Specifically, to further improve the accuracy of audio-visual synchronization and ensure synchronization effects, the time difference between the initial speech time difference and the end speech time difference are also recorded. Furthermore, the end speech time difference is added as part of the generated synchronization time difference, based on the initial speech time difference. The end speech time difference is the time difference between the last speech of the target object in the first video data and the second audio data.

[0103] If the difference between the initial speech time difference and the end speech time difference is within the error range (25ms), it indicates that the IPC's first video data and the microphone's second audio data correspond, and the difference is updated to a new synchronization time difference. If it is not within the error range, it indicates that the IPC's first video data and the microphone's second audio data do not correspond, and the cached first audio data and first video data are encoded together and then stored.

[0104] This embodiment utilizes the initial speech time difference and the end speech time difference to adjust the synchronization time difference, further improving the accuracy of audio-visual synchronization and ensuring synchronization effect.

[0105] The present embodiment will now be described and illustrated through preferred embodiments.

[0106] Figure 6 This is a flowchart of the audio-visual synchronization method according to a preferred embodiment. In this preferred embodiment, the recording device in the audio-visual synchronization system is a network video recorder (NVR); the camera device connected to the recording device is a network camera (IPC); the audio-visual synchronization method includes the following steps:

[0107] Step 1: Connect the NVR to the IPC and the audio pickup device, and set it to replace the first audio data in the audio and video data with the second audio data from the audio pickup device.

[0108] Step 2: The NVR acquires the audio and video data captured by the IPC and the second audio data recorded by the audio pickup device; it adds the NVR's recording timestamp to ensure that the timestamps in the audio and video data and the second audio data are synchronized, and subsequent calculations are based on the recording timestamp.

[0109] Step 3: After the NVR obtains the audio and video data of the IPC, it first determines whether the audio pickup device has an audio stream of the second audio data. If the audio pickup device has an audio stream, the audio and video data of the IPC are separated to obtain the first video data and the first audio data. If the audio pickup device does not have an audio stream, the audio and video data of the IPC are not separated (archived audio and video data), and the process proceeds directly to step 16.

[0110] Step 4: After the NVR obtains the second audio data from the sound pickup device, first determine whether the IPC has an audio / video stream. If the IPC has an audio / video stream, proceed to step 5; if the IPC does not have an audio / video stream, the audio / video data is a black screen. Continue to overlay the audio stream of the second audio data to obtain the archived audio / video data, and proceed to step 16.

[0111] Step 5: Determine whether the first and second conditions are met. The first condition is whether the NVR is acquiring the first video data and the second audio data for the first time (the NVR has just been powered on). The second condition is whether the audio pickup device or IPC has been disconnected and reconnected. If at least one condition is met, proceed to step 6. If neither condition is met, proceed to step 13.

[0112] Step 6, under the conditions of: power-on and / or disconnection and reconnection: At this time, set up a buffer area to cache the second audio data of the pickup device, the first video data of the IPC, and the first audio data of the IPC (for exception handling). The buffer time can be set to 60s, 70s, etc., and there is no limit to the buffer time.

[0113] Step 7: The video recognition algorithm in the video analyzer analyzes the first video data of the IPC to obtain the first time interval corresponding to each speech. The first timestamp is determined by the third and fourth time intervals. Specifically: Analyze whether the mouth of the person (target object) is moving. If the mouth changes from still to moving and the time exceeds the first set value (the first set value can be 100ms, which can be adjusted according to the application scenario), then record the start of a speech and record the corresponding third timestamp; thus, the third timestamps t1vb, t2vb, ... corresponding to each start of speech can be obtained; if the mouth changes from moving to still and the time exceeds the second set value (the second set value can be 100ms, which can be adjusted according to the application scenario), then record the end of a speech and record the corresponding fourth timestamp; thus, the fourth timestamps t1ve, t2ve, ... corresponding to each end of speech can be obtained (where: t represents time, v represents video analysis, b represents start, and e represents end).

[0114] Step 8: The audio recognition algorithm in the audio analyzer analyzes the second audio data from the sound pickup device. The second timestamp is determined by the fifth and sixth time intervals. Specifically: it analyzes whether there is a human voice (the target object). If the voice appears from silence and the time exceeds the third preset value (the third preset value can be 100ms, which can be adjusted according to the application scenario), then a speech start is recorded, and the corresponding fifth timestamp is recorded, thus obtaining the fifth timestamps t1ab, t2ab... for each speech start. If the voice disappears from silence and the time exceeds the fourth preset value (the fourth preset value can be 100ms, which can be adjusted according to the application scenario), then a speech end is recorded, and the corresponding sixth timestamp is recorded, thus obtaining the sixth timestamps t1ae, t2ae... for each speech start (Note: t represents time, a represents audio analysis, b represents start, and e represents end).

[0115] Step 9: Analyze the intervals between the timestamps recorded in Step 7 (i.e., the duration of each speech segment), and denote the first time interval as t1ve-t1vb = Δt1v, t2ve-t2vb = Δt2v...

[0116] Step 10: Analyze the intervals between the timestamps recorded in Step 8 (i.e., the duration of each speech segment), and denote the second time interval as t1ae-t1ab=Δt1a, t2ae-t2ab=Δt2a...

[0117] Step 11: Using Δt1v, Δt2v, Δt1a, and Δt2a obtained in steps 9 and 10, determine whether the first video data of the IPC corresponds to the second audio data of the audio pickup device, that is, determine whether the sound in the second audio data is spoken by the person in the video frame of the current first video data. Specifically: if the first time interval and the second time interval are all equal (Δt1v = Δt1a, Δt2v = Δt2a...), it can be considered that the duration of each speech segment in the first video data is the same as the duration of each speech segment in the second audio data. At this time, it can be concluded that the sound in the second audio data is spoken by the person in the video frame of the current first video data. Furthermore, a certain error is allowed between the first time interval and the second time interval; this error can be taken as 25ms.

[0118] Step 12: If the first video data of the IPC corresponds to the second audio data of the microphone, record the time when the first speech begins in the first video data of the IPC and the time when the first speech begins in the second audio data of the microphone, and the difference between the two: initial speech time difference ΔT = t1vb - t1ab, and then proceed to step 13; If the first video data of the IPC does not correspond to the second audio data of the microphone, it means that the sound collected by the microphone is not spoken by the person in the current video frame. In this case, use the first audio data and the first video data cached in step 6 to encode and store them together, and then proceed to step 7 to continue the analysis process.

[0119] Step 13: If the NVR is continuously streaming normally (acquiring audio and video data and second audio data), then the caching in step 6 can be skipped. Instead, real-time analysis can be performed, analyzing the second audio data from the audio pickup device and the first video data from the IPC, including the speech start timestamps tvb and tab, as well as the speech end timestamps tve and tae.

[0120] Step 14: Calculate the first time interval ΔTv = tve - tvb and the second time interval ΔTa = tae - tab. If the difference between the first and second time intervals ΔTv - ΔTa is within the error (25ms) of step 11, it indicates that the first video data of the IPC and the second audio data of the pickup device correspond. Update the synchronization time difference ΔT = tvb - tab and proceed to step 15. If it is not within the error range, it indicates that the first video data of the IPC and the second audio data of the pickup device do not correspond. Switch to using the first audio data and the first video data cached in step 6, encode them together, and store them. Proceed to step 16.

[0121] Step 15, as follows Figure 7 As shown, in practical applications, the frame rates of video frames and audio frames are different; video frames and audio frames do not correspond one-to-one; therefore, video frames are used as the time reference during decoding. Under this premise, to achieve audio-visual synchronization, firstly, the timestamp of the video frame of the first video data remains unchanged, and then the timestamp of the audio frame of the second audio data is added to the synchronization time difference ΔT calculated in step 12 or step 14. Encoding audio and video according to this rule ensures audio-visual synchronization and improves reliability; then proceed to Step 16 to save the data.

[0122] Step 16: Save the encoded target audio and video data or archived audio and video data to disk for easy retrieval and playback later.

[0123] This embodiment, through the aforementioned device, solves the problem of unstable video frame rates, leading to video skipping or stuttering. It improves the reliability of audio-visual synchronization by using the recording timestamp as the synchronization time basis, and completes audio-visual synchronization based on the first video data according to the first and second time intervals, thus improving the presentation quality of the synchronized video. Furthermore, it does not rely on time synchronization between devices, resulting in better reliability and applicability to audio pickup devices that do not support time synchronization. Switching schemes for different network environments are also provided.

[0124] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0125] This embodiment also provides an audio-visual synchronization device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," and "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0126] Figure 8 This is a structural block diagram of the audio-visual synchronization device of this embodiment, applied to an audio-visual synchronization system; the audio-visual synchronization system includes a recording device, an audio pickup device connected to the recording device, and a camera device; the device is suitable for recording devices, such as... Figure 8 As shown, the device includes: an acquisition module 210, a first analysis module 220, a second analysis module 230, and a synchronization module 240;

[0127] The acquisition module 210 is used to acquire first video data from the camera device and second audio data from the audio pickup device; the first video data and the second audio data are appended with the recording timestamp of the recording device;

[0128] The first analysis module 220 is used to analyze the first video data based on the recording timestamp to obtain the first time interval corresponding to each speech;

[0129] The second analysis module 230 is used to analyze the second audio data based on the video recording timestamp to obtain the second time interval corresponding to each speech.

[0130] The synchronization module 240 is used to synchronize the second audio data into the first video data according to the first time interval and the second time interval to obtain the target audio and video data.

[0131] The above-mentioned device solves the problem of unstable video frame rate, which causes video to jump or stutter. It uses the recording timestamp as the synchronization time basis to improve the reliability of audio-visual synchronization. It also uses the first video data as the video basis for synchronization based on the first time interval and the second time interval to complete the audio-visual synchronization and improve the presentation effect of the synchronized video.

[0132] In some embodiments, the acquisition module 210 is further configured to acquire audio and video data from the camera device and second audio data from the sound pickup device, and add a recording timestamp from the recording device to the audio and video data and the second audio data;

[0133] The audio and video data are stripped to obtain the first video data and the first audio data.

[0134] In some embodiments, the audio-visual synchronization device further includes: a judgment module;

[0135] The judgment module is also used to determine whether the first condition and the second condition are met after obtaining the first video data and the first audio data; the first condition is whether the recording device is acquiring the first video data and the second audio data for the first time; the second condition is whether the audio pickup device or the camera device has disconnected and reconnected.

[0136] If at least one condition is met, the first video data, the first audio data, and the second audio data are cached in the cache area;

[0137] If neither of the two conditions is met, then the first video data and the second audio data are analyzed in real time.

[0138] In some of these embodiments, the first analysis module 220 is further configured to use the pauses in the target object’s speech in the first video data as anchor points and the recording timestamp as a basis to perform speech analysis on the first video data to obtain the first timestamp of each speech.

[0139] Record the time interval between adjacent first timestamps to obtain the first time interval corresponding to each speech.

[0140] In some embodiments, the second analysis module 230 is further configured to use the pauses in the target object's speech in the second audio data as anchor points and, based on the video recording timestamps, perform speech analysis on the second audio data to obtain the second timestamp for each speech.

[0141] Record the time interval between adjacent second timestamps to obtain the second time interval corresponding to each speech.

[0142] In some embodiments, the synchronization module 240 is further configured to determine whether the target objects in the first video data and the second audio data correspond based on the first time interval and the second time interval;

[0143] If the target object in the first video data and the second audio data does not correspond, the first video data and the acquired first audio data are encoded and stored.

[0144] If the target objects in the first video data and the second audio data correspond, the initial speech time difference is recorded and used as the synchronization time difference; the initial speech time difference is the time difference between the first speech of the target object in the first video data and the second audio data.

[0145] Keeping the recording timestamp of the first video data unchanged, the recording timestamp of the second audio data is superimposed with the synchronization time difference to obtain the target audio and video data.

[0146] In some embodiments, the audio-visual synchronization device further includes: an adjustment module;

[0147] The adjustment module is used to record the end speech time difference in addition to the initial speech time difference; the end speech time difference is the time difference between the last speech of the target object in the first video data and the second audio data.

[0148] Adjust the synchronization time difference based on the initial speech time difference and the end speech time difference.

[0149] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0150] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0151] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0152] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0153] S1, acquire the first video data from the camera device and the second audio data from the audio pickup device; add the recording timestamp from the recording device to the first video data and the second audio data;

[0154] S2, based on the recording timestamp, analyze the first video data to obtain the first time interval corresponding to each speech;

[0155] S3, based on the video timestamp, analyze the second audio data to obtain the second time interval corresponding to each speech;

[0156] S4. Based on the first time interval and the second time interval, synchronize the second audio data into the first video data to obtain the target audio and video data.

[0157] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0158] Furthermore, in conjunction with the audio-visual synchronization methods provided in the above embodiments, this embodiment can also provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the audio-visual synchronization methods described in the above embodiments.

[0159] It should be noted that all information and data involved in this application are authorized by the user or fully authorized by all parties and will be used legally.

[0160] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0161] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0162] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for synchronizing audio and video, characterized in that, The method is applied to an audio-video synchronization system, the audio-video synchronization system comprises a recording device, a sound pickup device connected with the recording device and a camera device, the method is applied to the recording device, and the method comprises the following steps: obtaining first video data of the camera device and second audio data of the sound pickup device; the first video data and the second audio data are added with a recording time stamp of the recording device; based on the recording time stamp, analyzing the first video data to obtain a first time interval corresponding to each speech; based on the recording time stamp, analyzing the second audio data to obtain a second time interval corresponding to each speech; synchronizing the second audio data into the first video data according to the first time interval and the second time interval to obtain target audio-video data.

2. The method of claim 1, wherein, obtaining first video data of the camera device and second audio data of the sound pickup device comprises: obtaining audio-video data of the camera device and second audio data of the sound pickup device, and adding a recording time stamp of the recording device to the audio-video data and the second audio data; performing stripping processing on the audio-video data to obtain first video data and first audio data.

3. The method of claim 2, wherein, The method further comprises: after obtaining the first video data and the first audio data, judging whether a first condition and a second condition are met; the first condition is whether the recording device obtains the first video data and the second audio data for the first time; the second condition is whether the sound pickup device or the camera device is disconnected and reconnected; if at least one condition is met, the first video data, the first audio data and the second audio data are cached to a cache area; if neither of the two conditions is met, the first video data and the second audio data are analyzed in real time.

4. The method of claim 1, wherein, based on the recording time stamp, analyzing the first video data to obtain a first time interval corresponding to each speech comprises: taking a pause of a target object speaking in the first video data as an anchor point, and based on the recording time stamp, performing speech analysis on the first video data to obtain a first time stamp of each speech; recording a time interval between adjacent first time stamps to obtain a first time interval corresponding to each speech.

5. The method of claim 1, wherein, based on the recording time stamp, analyzing the second audio data to obtain a second time interval corresponding to each speech comprises: taking a pause of a target object speaking in the second audio data as an anchor point, and based on the recording time stamp, performing speech analysis on the second audio data to obtain a second time stamp of each speech; recording a time interval between adjacent second time stamps to obtain a second time interval corresponding to each speech.

6. The method of claim 1, wherein, synchronizing the second audio data into the first video data according to the first time interval and the second time interval to obtain target audio-video data comprises: determining whether target objects in the first video data and the second audio data correspond according to the first time interval and the second time interval. If the target object in the first video data and the second audio data do not correspond, the first video data and the acquired first audio data are encoded and stored; If the target object in the first video data and the second audio data correspond, an initial speech time difference is recorded, and the initial speech time difference is taken as a synchronization time difference; the initial speech time difference is a time difference of first speech of the target object in the first video data and the second audio data; The recording time stamp of the first video data is kept unchanged, and the recording time stamp of the second audio data is superimposed with the synchronization time difference to obtain target audio-video data.

7. The method of claim 6, wherein, The method further comprises: When the initial speech time difference is recorded, a final speech time difference is also recorded; the final speech time difference is a time difference of last speech of the target object in the first video data and the second audio data; The synchronization time difference is adjusted according to the initial speech time difference and the final speech time difference.

8. An audio-visual synchronization apparatus, characterized by comprising: The device is applied to an audio-video synchronization system; the audio-video synchronization system comprises a recording device, a sound pickup device connected with the recording device, and a camera device; the device is applied to the recording device, and the device comprises an acquisition module, a first analysis module, a second analysis module, and a synchronization module; The acquisition module is used to acquire first video data of the camera device and second audio data of the sound pickup device; the first video data and the second audio data are added with a recording time stamp of the recording device; The first analysis module is used to analyze the first video data based on the recording time stamp to obtain a first time interval corresponding to each speech; The second analysis module is used to analyze the second audio data based on the recording time stamp to obtain a second time interval corresponding to each speech; The synchronization module is used to synchronize the second audio data into the first video data according to the first time interval and the second time interval to obtain target audio-video data. 9.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute the steps of the audio-video synchronization method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the audio-video synchronization method in any one of claims 1 to 7.

Images