Audio and video synchronization method and device, equipment and storage medium

Abstract

The invention discloses an audio and video synchronization method and device, equipment and a storage medium, and relates to the technical field of audio and video, and the method comprises the steps: obtaining a first audio and a first video; based on the lip image in the first video, determining a first text for representing the speaking content of the speaker and the time of the first text; based on the first audio, determining a second text and time of the second text, the second text being semantic information of the first audio; when the similarity between the first text and the second text is higher than a preset similarity, determining a time difference which is a difference value between the time of the first text and the time of the second text; and if the time difference is greater than or equal to the preset duration, adjusting the starting moment of the first audio or the starting moment of the first video to realize synchronization of the first audio and the first video. Through the method, audio and video synchronization can be realized without manual intervention, and the audio and video synchronization efficiency and accuracy are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio and video technology, and in particular to an audio and video synchronization method, device, equipment and storage medium. Background Art

[0002] In teaching, lectures, or meetings, recorders often deploy multiple acquisition devices to collect audio and video of users' speeches from multiple perspectives to record the user's speech process in all directions. Due to the different performance of different acquisition devices, the time spent on processing audio or video is also different. Therefore, it is easy to cause the collected audio and video to be out of sync.

[0003] To achieve audio and video synchronization, in related technologies, when recording the user's speech process through different acquisition devices, the recorder often manually adjusts the parameters of different acquisition devices (such as acquisition rate, acquisition time, location, etc.) based on their own experience to synchronize the audio and video collected by different acquisition devices.

[0004] However, in actual applications, the deployment of application scenarios and acquisition equipment often changes. When the deployment of application scenarios and acquisition equipment often changes, in order to achieve audio and video synchronization, the recorder needs to frequently adjust the parameters of the acquisition equipment in the related technology. In the case where the parameters of the acquisition equipment need to be adjusted frequently, the method of manually adjusting the parameters of the acquisition equipment to achieve audio and video synchronization depends on the experience of the recorder, and may not be able to respond to the frequently changing situation in a timely manner, and it is difficult to ensure accuracy, which will affect the efficiency and accuracy of audio and video synchronization. Summary of the invention

[0005] The embodiments of the present application provide an audio and video synchronization method, apparatus, device and storage medium. The embodiments of the present application can achieve audio and video synchronization without human intervention, so as to improve the efficiency and accuracy of audio and video synchronization.

[0006] In a first aspect, an embodiment of the present application provides an audio and video synchronization method, including:

[0007] Acquire a first audio and a first video, where the first video includes a lip image, and the lip image is used to represent the lip movement of the speaker;

[0008] Based on the lip image in the first video, determine a first text and a time of the first text; the first text is used to represent the content of the speaker's speech;

[0009] Based on the first audio, determine a second text and a time of the second text, where the second text is semantic information of the first audio;

[0010] When the similarity between the first text and the second text is higher than a preset similarity, determining a time difference, where the time difference is a difference between the time of the first text and the time of the second text;

[0011] If the time difference is greater than or equal to the preset duration, the start time of the first audio or the start time of the first video is adjusted to achieve synchronization of the first audio and the first video.

[0012] Optionally, the determining of the first text and the time of the first text based on the lip image in the first video includes:

[0013] The first video is input into a preset lip shape detection model, and lip shape recognition is performed on the lip image in the first video to obtain the first text and the time of the first text.

[0014] Optionally, the determining of the second text and the time of the second text based on the first audio includes:

[0015] The first audio is input into a preset speech recognition model, audio recognition is performed on the first audio, and a second text and the time of the second text are obtained.

[0016] Optionally, the obtaining of the first audio and the first video includes:

[0017] The video and audio are collected in real time to obtain a first audio and a first video.

[0018] Optionally, the obtaining of the first audio and the first video includes:

[0019] Get the target audio and video;

[0020] Parsing the target audio and video to obtain audio data and video data; the audio data includes at least one audio segment, and the video data includes at least one video segment;

[0021] A first audio in the audio data is obtained, and a first video in the video data is obtained; a difference between a timestamp of the first audio and a timestamp of the first video is less than a threshold.

[0022] Optionally, the audio and video synchronization method further includes:

[0023] The similarity between the first text and the second text is calculated by a text similarity algorithm.

[0024] In a second aspect, an embodiment of the present application provides an audio and video synchronization device, including:

[0025] A data acquisition module, used to acquire a first audio and a first video, wherein the first video includes a lip image, and the lip image is used to represent the lip movement of the speaker;

[0026] A video text acquisition module, used to determine a first text and a time of the first text based on a lip image in a first video; the first text is used to represent the content of a speaker's speech;

[0027] An audio-text acquisition module, used to determine a second text and a time of the second text based on the first audio, wherein the second text is semantic information of the first audio;

[0028] A similarity calculation module, used to calculate the similarity between the first text and the second text by using a text similarity algorithm;

[0029] A time difference acquisition module, used to determine a time difference when the similarity between the first text and the second text is higher than a preset similarity, the time difference being the difference between the time of the first text and the time of the second text;

[0030] A synchronization judgment module is used to determine the relationship between the time difference and the preset duration;

[0031] The synchronization adjustment module is used to adjust the starting time of the first audio or the starting time of the first video if the time difference is greater than or equal to a preset duration, so as to achieve synchronization between the first audio and the first video.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing program instructions, wherein the processor is configured to execute the audio and video synchronization method as described in the first aspect or any optional method when running the program instructions.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, it implements the audio and video synchronization method as described in the first aspect or any optional method.

[0034] From the above, it can be seen that the audio and video synchronization method provided in the embodiment of the present application determines the first text (i.e., the text representing the content of the speaker's speech) and the time corresponding to the first text based on the lip image in the first video after acquiring the first audio and the first video (including the lip image of the speaker) during the speaker's speech; determines the second text (i.e., the semantic information of the first audio) and the time corresponding to the second text based on the first audio; when the similarity between the first text and the second text is higher than the preset similarity, the time difference between the time of the first text and the time of the second text is also determined; if the time difference is greater than or equal to the preset duration, the first audio or the first video is adjusted in time so that the image in the first video and the sound in the first audio can be kept consistent in time, thereby achieving audio and video synchronization. If the similarity between the first text and the second text is higher than the preset similarity, it can be considered that the first text and the second text correspond to the same speech content of the speaker, so that the first audio and the first video can correspond to the same speech content. On the basis of the same content, the time difference is also taken into consideration, and the starting time of the first audio or the first video is adjusted according to the time difference. In this way, the audio and video synchronization method provided in the embodiment of the present application can effectively ensure the accuracy of the synchronized content, ensure the synchronization effect, and effectively improve the synchronization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 is a flow chart of an audio and video synchronization method provided in an embodiment of the present application;

[0037] Figure 2 is a schematic diagram of an audio and video synchronization method provided in an embodiment of the present application;

[0038] Figure 3 is a schematic diagram of an embodiment of the present application applied to a real-time recording scenario;

[0039] Figure 4 It is a schematic diagram of an embodiment of the present application being applied to later editing of a recorded file;

[0040] Figure 5 is a schematic diagram of an audio and video synchronization device provided in an embodiment of the present application;

[0041] Figure 6 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0043] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0044] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0045] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0046] In teaching, lectures, or meetings, recorders often deploy multiple acquisition devices to collect audio and video of users' speeches from multiple perspectives to record the user's speech process in all directions. Due to the different performance of different acquisition devices, the time spent on processing audio or video is also different. Therefore, it is easy to cause the collected audio and video to be out of sync.

[0047] The following example illustrates the process of recording a user's speech through multiple acquisition devices. For example, in a conference room, audio and video recording of a user's speech is performed, and the acquisition devices required include an audio acquisition device and a video acquisition device. The video acquisition device includes a video acquisition device A and a video acquisition device B at different viewing angles. The video acquisition device A and the video acquisition device B can be used to record different viewing angles of the speaker. During the recording process, in order to record the user's speech process in all directions, it is necessary to switch between the video acquisition device A and the video acquisition device B to change the shooting perspective. However, due to the different performances between the video acquisition device A and the video acquisition device B, the time consumed to process the video will also change after the viewing angle is switched, which can easily lead to the recorded audio and video being out of sync.

[0048] To achieve audio and video synchronization, in related technologies, when recording the user's speech process through different acquisition devices, the recorder often manually adjusts the parameters of different acquisition devices (such as acquisition rate, acquisition time, location, etc.) based on their own experience to synchronize the audio and video collected by different acquisition devices.

[0049] However, in actual applications, application scenarios and the deployment of acquisition equipment often change. When application scenarios and the deployment of acquisition equipment often change, in order to achieve audio and video synchronization in related technologies, recorders need to frequently adjust the parameters of the acquisition equipment. Frequent manual adjustment of the parameters of the acquisition equipment by recorders will consume a lot of time, resulting in the inability to respond to changes in deployment in a timely manner, thereby affecting the efficiency of audio and video synchronization; and manual adjustment of the parameters of the acquisition equipment is too dependent on the experience and skills of the recorders, resulting in unstable accuracy of the parameter adjustment of the acquisition equipment, thereby affecting the accuracy of audio and video synchronization.

[0050] Based on this, the embodiments of the present application provide an audio and video synchronization method, apparatus, device and storage medium. In the embodiments of the present application, the text in the audio and video and the time corresponding to the text can be obtained based on the audio of the speaker's speech and the video containing lip movements. The audio or video can be adjusted in time based on the text in the audio and video and the corresponding time. Audio and video synchronization can be achieved without human intervention, thereby improving the efficiency and accuracy of audio and video synchronization.

[0051] Figure 1 The audio and video synchronization method provided by the embodiment of the present application is shown. Figure 1 As shown, the method includes:

[0052] S1, obtaining a first audio and a first video, where the first video includes a lip image, and the lip image is used to represent the lip movements of a speaker.

[0053] Specifically, during the real-time recording of audio and video or the post-editing of the packaged recording file, audio clips (i.e., first audio) and video clips (i.e., first video) are continuously obtained. The duration of the first audio is equal to the duration of the first video, the first video includes a lip image for representing the lip movements of the speaker, and the first audio includes the speaker's voice.

[0054] It should be noted that the first audio and the first video obtained at the same time must correspond to the same speaker.

[0055] S2, determining a first text and the time of the first text based on the lip image in the first video; the first text is used to represent the content of the speaker's speech.

[0056] Specifically, this step identifies the first video containing the speaker's lip image, determines the text information (i.e., the first text) corresponding to the speaker's lip movement, that is, obtains the content of the speaker's speech in the video, and determines the time of the first text based on the timestamp of the first video.

[0057] S3, based on the first audio, determine a second text and the time of the second text, the second text being semantic information of the first audio.

[0058] Specifically, this step recognizes the first audio containing the speaker's voice, and can convert the audio information during the speaker's speech into semantic information, that is, the second text, and then obtain the content of the speaker's speech in the audio, and the time of the second text can be determined based on the timestamp of the first audio.

[0059] S4, when the similarity between the first text and the second text is higher than a preset similarity, determining a time difference, where the time difference is a difference between the time of the first text and the time of the second text.

[0060] Specifically, this step first compares the similarity between the first text and the second text. If the similarity between the first text and the second text is higher than the preset similarity, it means that the first text and the second text match and correspond to the same speech content of the speaker recorded. On this basis, by calculating the time difference between the time of the first text and the time of the second text, the time difference between the first audio and the first video can be obtained.

[0061] S5: If the time difference is greater than or equal to the preset duration, adjust the starting time of the first audio or the starting time of the first video to achieve synchronization between the first audio and the first video.

[0062] The time difference is the time deviation between the first audio and the first video. When the time deviation is too large (i.e., greater than or equal to a preset duration), it can be considered that the first audio and the first video are not synchronized. In the case where the first audio and the first video are not synchronized, the first audio or the first video is adjusted to complete the synchronization of the first audio and the first video.

[0063] By using the audio and video synchronization method provided in the embodiment of the present application, the first text (i.e., the text representing the speaker's speech content) and the time corresponding to the first text can be determined from the first video according to the speaker's lip movements; the second text (i.e., the semantic information of the first audio) and the time corresponding to the second text can be determined from the first audio; when the similarity between the first text and the second text reaches the preset similarity, it is considered that the first text and the second text correspond to the same speech content of the speaker, and on this basis, the time difference between the time of the first text and the time of the second text is determined. If the time difference is greater than or equal to the preset duration, the first audio or the first video is adjusted in time according to the time difference, so that the image in the first video and the sound in the first audio can be consistent in time, thereby achieving the purpose of audio and video synchronization. If the similarity between the first text and the second text is higher than the preset similarity, it can be considered that the first text and the second text correspond to the same speech content of the speaker, so that the first audio and the first video can correspond to the same speech content. On the basis of the same content, the time difference is also taken into consideration, and the starting time of the first audio or the first video is adjusted according to the time difference. In this way, the audio and video synchronization method provided in the embodiment of the present application can effectively ensure the accuracy of the synchronized content, ensure the synchronization effect, and effectively improve the synchronization efficiency.

[0064] It should be noted that if the first audio is an audio segment in the audio data of the recorded file, and the first video is a video segment in the video data of the recorded file, the synchronization device repeats the above steps S1-S5 until all acquired audio and video are synchronized.

[0065] Next, the above steps S1 to S5 are described.

[0066] In step S1, the first audio and the first video obtained may be obtained from audio and video collected in real time, or may be obtained from a pre-recorded and packaged recording file.

[0067] In one embodiment, the process of obtaining the first audio and the second audio from the audio and video collected in real time can be applied to the scenario of real-time recording of the speaker's speech process. In this scenario, the audio during the speaker's speech is collected in real time by an audio acquisition device, and the first audio is obtained therefrom. The video during the speaker's speech is collected in real time by a video acquisition device, and the first video is obtained therefrom. The audio acquisition device can be a microphone, and the video acquisition device can be a camera. Among them, the acquisition time corresponding to the first audio and the first video cannot deviate too much, and the duration of the first audio and the first video is also preferably consistent. For example, the first audio can be an audio clip with a duration of t collected at the current moment, and the first video can be a video clip with a duration of t collected at the current moment, and in the process of real-time recording of audio and video, audio clips and video clips are continuously acquired, that is, the first audio and the first video are continuously acquired.

[0068] It should be noted that since the first audio and the first video are respectively captured through different acquisition devices, or captured through different audio acquisition modules and video acquisition modules in the same acquisition device, even if the acquisition time corresponding to the first audio and the first video is the same, due to the performance of different acquisition devices or different modules of the same acquisition device, different time may be consumed during the encoding conversion of the first audio and the first video, thereby causing the first audio and the first video to be out of sync.

[0069] It should also be noted that in the process of real-time recording, it may be necessary to shoot different people or different angles of the same person, that is, the capture process of the first video may involve multiple camera screen switching, but when the first audio is captured, only one audio source can be input at the same time. That is, during the entire recording process, images can be captured from different perspectives of the same speaker, or the sounds and images of different speakers can be captured at different time periods, but the first video and the first audio must correspond to the same speaker, and the first video must at least include the speaker's lip image.

[0070] It is worth mentioning that when processing audio and video, the problem of audio and video being out of sync is prone to occur not only in the process of recording the video when the speaker is speaking, but also in the process of video editing after recording.

[0071] Next, the process of obtaining the first audio and the first video from a pre-recorded and packaged recording file during the video editing process will be described.

[0072] In one embodiment, for an application scenario in which a recorded file is edited, obtaining the first audio and the first video includes: obtaining the target audio and video; parsing the target audio and video to obtain audio data and video data; the audio data includes at least one audio segment, and the video data includes at least one video segment; obtaining the first audio in the audio data, and obtaining the first video in the video data; the difference between the timestamp of the first audio and the timestamp of the first video is less than a threshold.

[0073] In this embodiment, the target audio and video is a recorded file that has been packaged. The target audio and video can be in ts format or mp4 format.

[0074] In order to obtain the original audio data and video data that can be analyzed, the target audio and video need to be parsed. The parsing process includes: first, decapsulating the target audio and video and separating them into audio compression coded data and video compression coded data, and then decoding them into PCM format audio data and RGB format video data through an audio decoder and a video decoder respectively. The obtained PCM format audio data and RGB format video data are audio data and video data that can be analyzed.

[0075] In this embodiment, in order to facilitate the video editing operation, the audio data and the video data are segmented, and the segmented audio data includes at least one audio segment, and the segmented video data includes at least one video segment. Further, the first audio in the audio data is obtained, that is, an audio segment is obtained from at least one audio segment of the audio data as the first audio, and the first video in the video data is obtained, that is, a video segment is obtained from at least one video segment of the video data as the first video. In the process of obtaining the first audio and the first video, considering that the time deviation of the first audio and the first video is too large, the first audio and the first video may not correspond to the speech process of the speaker in the same period, so there is no need to synchronize the first audio and the first video. In order to avoid performing unnecessary synchronization processes, when obtaining the first audio and the first video, it is necessary to ensure that the first audio and the first video roughly match the speech process of the speaker in the same period, that is, to ensure that the time deviation between the first audio and the first video is not too large. Specifically, you can refer to the difference between the start timestamp of the first audio and the start timestamp of the first video to obtain the first audio and the first video. If the start timestamp of the first audio and the start timestamp of the first video are less than the threshold, it means that the time deviation between the first audio and the first video is not large. Therefore, it is believed that the first audio and the first video may correspond to the speech of the speaker in the same period of time.

[0076] By adopting this solution, when obtaining the first audio and the first video, not only can the first audio and the first video be obtained in real time in the scene where the recording terminal records audio and video in real time, but also the audio and video to be edited (i.e., the target audio and video) can be parsed in the audio and video editing scene to obtain the first audio and the first video. In this way, the audio and video synchronization method provided in the embodiment of the present application can be implemented in a variety of application scenarios to meet the needs of users to perform audio and video synchronization in different scenarios.

[0077] In the above step S2, based on the lip image in the first video, the first text and the time of the first text are determined, specifically including: inputting the first video into a preset lip shape detection model, performing lip shape recognition on the lip image in the first video, and obtaining the first text and the time of the first text.

[0078] Specifically, the first video contains a lip image of the speaker. By performing lip shape recognition on the first video through a preset lip shape detection model, the content of the speaker's speech, that is, the first text, can be identified from the speaker's lip movements. At the same time, the lip shape detection model can also determine the timestamp of the first text based on the timestamp of the video frame in which the speaker speaks in the first video, thereby determining the time of the first text. The time of the first text refers to the interval between the start timestamp and the end timestamp in the timestamp of the first text, that is, the time from the start to the end of the speaker's speech in the first video.

[0079] In the above step S3, based on the first audio, determining the second text and the time of the second text specifically includes: inputting the first audio into a preset speech recognition model, performing audio recognition on the first audio, and obtaining the second text and the time of the second text.

[0080] Specifically, the first audio includes the sound of the speaker, and the preset speech recognition model (such as the ASR model) can automatically convert the speaker's sound information into semantic information, that is, the second text, and at the same time, the timestamp corresponding to the second text can be determined according to the timestamp of the first audio, thereby determining the time of the second text. Among them, the time of the second text refers to the interval between the start timestamp and the end timestamp in the timestamp of the second text, that is, the time from the start of the speaker's speech to the end of his speech in the first audio.

[0081] By adopting this solution, a lip shape detection model and a speech recognition model are used to determine the first text corresponding to the first video and the second text corresponding to the first audio, and the time corresponding to the first text and the second text is obtained. This can more accurately and quickly identify the speech content and speech time of the speaker in the audio and video.

[0082] In the above step S4, since the timestamp of the first video corresponds to the time of the first text, and the timestamp of the first audio corresponds to the time of the second text, the time difference between the time of the first text and the time of the second text can represent the time deviation between the first video and the first audio. In this embodiment, when calculating the time difference between the time of the first text and the time of the second text, it is necessary to determine whether the content of the first text and the content of the second text correspond to the same speech content. If they correspond to the same speech content, it means that the first audio and the first video correspond to the same speech content, so that the first audio and the first video have the necessary audio-visual synchronization.

[0083] In this embodiment, when judging whether the content of the first text and the content of the second text correspond to the same speech content, the similarity between the first text and the second text is compared with the preset similarity. Specifically, when the similarity between the first text and the second text is higher than the preset similarity, it indicates that the similarity between the first text and the second text is high, and it can be considered that the content of the first text and the content of the second text correspond to the same speech content, that is, the first audio and the first video correspond to the same speech content. When the similarity between the first text and the second text is lower than the preset similarity, it indicates that the similarity between the first text and the second text is low, and it can be considered that the content of the first text and the content of the second text do not correspond to the same speech content, that is, the first audio and the first video do not correspond to the same speech content, that is, the first audio and the first video do not need to be synchronized.

[0084] Furthermore, in this embodiment, the similarity between the first text and the second text may be calculated using a text similarity algorithm.

[0085] Optionally, the text similarity algorithm may be a cosine similarity algorithm. The process of calculating the similarity between the first text and the second text using the cosine similarity algorithm is as follows: the first text and the second text are converted into vectors respectively to obtain a first vector and a second vector; the cosine value of the angle between the first vector and the second vector is calculated using the cosine similarity formula; and the similarity between the first text and the second text is represented by the cosine value. The range of the cosine value is -1 to 1. The closer the cosine value is to 1, the higher the similarity between the first text and the second text.

[0086] By adopting this embodiment, a text similarity algorithm is used to confirm whether the first text and the second text correspond to the same speech content of the speaker, and whether the first audio and the first video correspond to the speech of the speaker in the same time period, and then it is determined whether the first audio and the first video need to be synchronized. If they do not correspond to the speech of the speaker in the same time period, there is no need for synchronization, and there is no need to adjust the time of the first audio or the first video.

[0087] Furthermore, in the above step S4, when the similarity between the first text and the second text is higher than the preset similarity, the specific process of determining the time difference is as follows:

[0088] The time difference is obtained by calculating the difference between the time of the first text and the time of the second text. The time difference is the time difference between the first audio and the first video, that is, the time difference between the sound and the picture of the speaker when speaking.

[0089] Specifically, the difference between the time of the first text and the time of the second text can be obtained by calculating the difference between the start timestamp in the time of the first text and the start timestamp in the time of the second text. The start timestamp in the time of the first text corresponds to the time when the speaker starts speaking in the first video, and the start timestamp in the time of the second text corresponds to the time when the speaker starts speaking in the first audio.

[0090] By adopting this scheme, obtaining the difference between the time of the first text and the time of the second text is equivalent to obtaining the time difference between the start of the speaker's speech in the first video and the first audio, and can accurately and effectively identify the time difference between the first audio and the first video.

[0091] In the above step S5, if the time difference is greater than or equal to the preset duration, the starting time of the first audio or the starting time of the first video is adjusted to achieve synchronization of the first audio and the first video, including: determining whether the time difference is greater than or equal to the preset duration.

[0092] Specifically, if the time difference is greater than or equal to the preset duration, the first audio and the first video are not synchronized. The preset duration is set by the user, for example, the preset duration can be set to -185 to +90ms. If the time difference is less than the preset duration, from the user's perspective, since the time difference is small, the user cannot feel the lack of synchronization between the audio and the picture, so it can be considered that the sound and the picture are synchronized, that is, the audio and the picture between the first audio and the first video are synchronized, and no adjustment is required. If the time difference is greater than or equal to the preset duration, since the time difference is large, the user can easily feel the lack of synchronization between the sound and the picture, so when the time difference between the first audio and the first video is large, it can be considered that there is a lack of synchronization between the audio and the picture between the first audio and the first video, and synchronization adjustment is required between the first audio and the first video.

[0093] By adopting this embodiment, by comparing the time difference with the preset duration, it is determined whether the audio and video have audio and video asynchronism that can be perceived by the user through the senses. If the user cannot perceive it, there is no need to adjust the time. Compared with manually judging whether time adjustment is needed, it is more reliable and accurate, and can ensure that the sound in the first audio and the picture in the first video are synchronized in the audience's perception.

[0094] Furthermore, in the case where the first audio and the first video are not synchronized, the process of performing audio and video synchronization includes: adjusting the starting time of the first audio or the starting time of the first video according to the time difference.

[0095] Specifically, when performing audio and video synchronization according to the time difference, you can choose to adjust the starting time of the first audio so that the starting time of the first audio is aligned with the starting time of the first video. In addition, because the human ear is more sensitive to audio than video, in this embodiment, you can also choose not to adjust the first audio, but to adjust the starting time of the first video so that the starting time of the first video is aligned with the starting time of the first audio, and this application does not limit this; wherein the starting time of the first video is the starting timestamp of the first video, and the starting time of the first audio is the starting timestamp of the first audio.

[0096] In one embodiment, when performing audio and video synchronization based on the time difference, in addition to adjusting the start time of the first audio or the start time of the first video, the synchronization of the first audio and the first video can also be achieved by adjusting the timestamp interval of the first audio or the timestamp interval of the first video. For example, assuming that the preset duration is -185 to +90ms, the specific steps for performing audio and video synchronization by adjusting the timestamp interval of the first video are as follows: When the time difference between the first audio and the first video is less than or equal to -185ms, it means that the first audio lags behind the first video too much. At this time, by increasing the timestamp interval of the first video, the first video can be slowed down to achieve synchronization with the first audio. When the time difference between the first audio and the first video is greater than or equal to 90ms, it means that the first audio is ahead of the first video. At this time, by reducing the timestamp interval of the first video, the first video can be fast-forwarded to achieve synchronization with the first audio.

[0097] The detailed implementation of the audio and video synchronization method is further described in detail below in conjunction with specific embodiments. Figure 2 yes Figure 1 The schematic diagram of the specific process of the embodiment shown is shown. Figure 2 The audio and video synchronization method of this embodiment includes the following steps:

[0098] Get a first audio and a first video, wherein the first audio is an audio clip with a duration of t, and the first video is a video clip with a duration of t; and the first audio is in a PCM format, and the first video is in an RGB format.

[0099] The first video is subjected to lip shape recognition through the lip shape detection model, and the first text txt1 and the time of the first text are recognized.<T1,T2> Among them, the start timestamp of the first text is T1, and the end timestamp is T2, that is, the time interval from the start of the speaker's speech to the end of the speech in the first video is<T1,T2> .

[0100] Perform audio recognition on the first audio through the speech recognition model, and recognize the second text txt2 and the time of the second text<T3,T4> The start timestamp of the second text is T3, and the end timestamp is T4, that is, the time interval from the start of the speaker's speech to the end of his speech in the first audio is<T3,T4> .

[0101] The first text and the second text are identified by text similarity. Specifically, the similarity between the first text and the second text can be calculated by a text similarity algorithm, and when the similarity between the first text and the second text is higher than a preset similarity, the time difference Tp is determined. For example, the time of the first text is<T1,T2> , the time of the second text is<T3,T4> , the time difference Tp is the difference between T3 and T1.

[0102] The first audio and the first video are compressed and encoded. The first video in RGB format is compressed and encoded into H264 format, and the first audio in PCM format is compressed and encoded into AAC format.

[0103] The starting time of the compressed and encoded first video or the first audio is adjusted according to the time difference Tp. After the adjustment, the first video and the first audio are packaged as audio and video to obtain a recording file with synchronized audio and video.

[0104] In the above embodiment, the first audio and the first video may be obtained from audio and video collected in real time in a real-time recording scenario, or may be obtained from a pre-recorded and packaged recording file in a post-editing scenario.

[0105] The following combination Figure 3 , Figure 4 , the audio and video synchronization methods in different application scenarios are described in detail.

[0106] Figure 3 Schematic diagram of an embodiment of the present application applied to a real-time recording scenario. Figure 3 The audio and video synchronization method of this embodiment includes the following steps:

[0107] In the real-time recording scenario, the speaker's speech is captured in real time through a video collector to obtain a video clip in NV12 format (i.e., the first video). In order to facilitate video recognition processing, the first video in NV12 format is converted to obtain the first video in RGB format. The first video in RGB format is input into a preset lip shape detection model, and the lip shape is recognized on the lip image in the first video to obtain the first text and the timestamp of the first text, thereby determining the time of the first text.

[0108] While the video collector is used to capture the video of the speaker's speech in real time, the audio collector is used to capture the audio of the speaker's speech in real time to obtain an audio clip in PCM format (i.e., the first audio), and the first audio in PCM format is input into the speech recognition model to obtain the second text and the timestamp of the second text, thereby determining the time of the second text.

[0109] In order to obtain a recording file for synchronous playback of audio and video, the first audio and the first video are encoded and packaged. The encoding process includes: encoding the collected first video in NV12 format into H264 format, and encoding the collected first audio in PCM format into AAC format. Optionally, before encoding the first audio in PCM format, the first audio is processed using a mixing algorithm and a 3A algorithm; wherein, processing the first audio with a mixing algorithm can make the sound source sound better; processing the first audio with a 3A algorithm can effectively remove noise existing in the first audio.

[0110] According to the difference between the time of the first text and the time of the second text, the starting time of the encoded first audio or first video is adjusted. After the adjustment is completed, the first audio and the first video are packaged to obtain a recording file with synchronized audio and video; wherein the recording file can be in ts format or mp4 format.

[0111] In an embodiment of the present application, the executor of the audio and video synchronization method can also be a recording terminal that can collect audio and video in real time and synchronize audio and video in real time. The recording terminal integrates an audio collector and a video collector, and the performance of the recording terminal must meet the requirements of running at least one visual algorithm and one audio algorithm, and can analyze visual data and audio data in real time. In other words, the time consumed to run the visual algorithm and the audio algorithm must be less than or equal to the actual recording time. For example, the time consumed for synchronizing one minute of audio and video must be less than one minute, so the recording terminal needs to have a processing speed of at least 20fps and 1080P, so that the recording terminal combines real-time recording and real-time analysis, and can perform real-time time correction.

[0112] Figure 4 is a schematic diagram of an embodiment of the present application for later editing of a recorded file, with reference to Figure 4 The audio and video synchronization method of this embodiment includes the following steps:

[0113] In the scenario of editing a recorded file, the recorded file in ts format or mp4 format is first decapsulated to obtain an audio stream and a video stream. Further, the video stream is decoded into video data in RGB format by a video decoder, and the audio stream is decoded into audio data in PCM format by an audio decoder, wherein the audio data includes at least one audio segment, and the video data includes at least one video segment.

[0114] A video clip (i.e., a first video) is obtained from the video data, and an audio clip (i.e., a first audio) is obtained from the audio data, wherein the difference between the timestamp of the first audio and the timestamp of the first video is less than a threshold value. The first video in RGB format is input into a preset lip shape detection model, and lip shape recognition is performed on the lip image in the first video to obtain a first text and a timestamp of the first text, thereby determining the time of the first text, and the first audio in PCM format is input into a speech recognition model to obtain a second text and a timestamp of the second text, thereby determining the time of the second text.

[0115] In order to obtain a recording file in which audio and video can be played synchronously, the first audio and the first video also need to be encoded and packaged. The encoding process includes: encoding the first video in RGB format into H264 format, and encoding the first audio in PCM format into AAC format.

[0116] According to the difference between the time of the first text and the time of the second text, the start time of the encoded first audio or first video is adjusted, and after the adjustment is completed, the first audio and the first video are audio and video packaged until all the video segments and audio segments in the video data and audio data are packaged to obtain a corrected recording file. The corrected recording file is also in ts format or mp4 format.

[0117] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0118] Figure 5 It is a schematic diagram of an audio and video synchronization device provided in an embodiment of the present application. The device 10 includes: a data acquisition module 101, a video text acquisition module 102, an audio text acquisition module 103, a similarity calculation module 104, a time difference acquisition module 105, a synchronization judgment module 106 and a synchronization adjustment module 107.

[0119] The data acquisition module 101 is used to acquire a first audio and a first video, wherein the first video includes a lip image, and the lip image is used to represent the lip movements of the speaker.

[0120] The video text acquisition module 102 is used to determine the first text and the time of the first text based on the lip image in the first video; the first text is used to represent the content of the speaker's speech.

[0121] The audio-text acquisition module 103 is used to determine a second text and a time of the second text based on the first audio, where the second text is semantic information of the first audio.

[0122] The similarity calculation module 104 is used to calculate the similarity between the first text and the second text by using a text similarity algorithm.

[0123] The time difference acquisition module 105 is used to determine the time difference when the similarity between the first text and the second text is higher than a preset similarity, where the time difference is the difference between the time of the first text and the time of the second text.

[0124] The synchronization determination module 106 is used to determine the size relationship between the time difference and the preset duration.

[0125] The synchronization adjustment module 107 is used to adjust the starting time of the first audio or the starting time of the first video when the time difference is greater than or equal to a preset time length, so as to achieve synchronization between the first audio and the first video.

[0126] Optionally, the data acquisition module 101 is specifically used to collect video and audio in real time to obtain a first audio and a first video.

[0127] Optionally, the data acquisition module 101 is also used to acquire target audio and video; parse the target audio and video to obtain audio data and video data; the audio data includes at least one audio segment, and the video data includes at least one video segment; acquire the first audio in the audio data, and acquire the first video in the video data; the difference between the timestamp of the first audio and the timestamp of the first video is less than a threshold.

[0128] Optionally, the video text acquisition module 102 is specifically configured to input the first video into a preset lip shape detection model, perform lip shape recognition on the lip image in the first video, and obtain the first text and the time of the first text.

[0129] Optionally, the audio-text acquisition module 103 is specifically configured to input the first audio into a preset speech recognition model, perform audio recognition on the first audio, and obtain the second text and the time of the second text.

[0130] The audio and video synchronization device provided in the embodiment of the present application is used to execute the audio and video synchronization method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0131] Combination Figure 6 As shown, an embodiment of the present application provides an electronic device 600, including a processor (processor) 601 and a memory (memory) 602. Optionally, the electronic device 600 may also include a communication interface (CommunicationInterface) 603 and a bus 604. Among them, the processor 601, the communication interface 603, and the memory 602 can communicate with each other through the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can call the logic instructions in the memory 602 to execute the audio and video synchronization method in the above embodiment.

[0132] In addition, the logic instructions in the above-mentioned memory 602 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. The memory 602, as a computer-readable storage medium, can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present application. The processor 601 executes functional applications and data processing by running the program instructions / modules stored in the memory 602, that is, implements the audio and video synchronization method in the above-mentioned embodiment. The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 602 may include a high-speed random access memory and may also include a non-volatile memory.

[0133] The embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the audio and video synchronization method in the above embodiment is implemented. The computer-readable storage medium can be a transient computer-readable storage medium or a non-transient computer-readable storage medium.

[0134] The above description and the accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them.

Claims

1. An audio and video synchronization method, characterized in that: include: Acquire a first audio and a first video, where the first video includes a lip image, and the lip image is used to represent the lip movement of the speaker; Determine a first text and a time of the first text based on the lip image in the first video; The first text is used to represent the content of the speaker's speech; Based on the first audio, determine a second text and a time of the second text, where the second text is semantic information of the first audio; When the similarity between the first text and the second text is higher than a preset similarity, determining a time difference, where the time difference is a difference between the time of the first text and the time of the second text; If the time difference is greater than or equal to a preset duration, the starting time of the first audio or the starting time of the first video is adjusted to achieve synchronization of the first audio and the first video.

2. The audio and video synchronization method according to claim 1, characterized in that: The determining the first text and the time of the first text based on the lip image in the first video includes: The first video is input into a preset lip shape detection model, and lip shape recognition is performed on the lip image in the first video to obtain the first text and the time of the first text.

3. The audio and video synchronization method according to claim 1, characterized in that: The determining, based on the first audio, a second text and a time of the second text comprises: The first audio is input into a preset speech recognition model, audio recognition is performed on the first audio, and the second text and the time of the second text are obtained.

4. The audio and video synchronization method according to any one of claims 1 to 3, characterized in that: The obtaining of the first audio and the first video includes: The video and audio are collected in real time to obtain the first audio and the first video.

5. The audio and video synchronization method according to any one of claims 1 to 3, characterized in that: The obtaining of the first audio and the first video includes: Get the target audio and video; Parsing the target audio and video to obtain audio data and video data; the audio data includes at least one audio segment, and the video data includes at least one video segment; The first audio in the audio data is obtained, and the first video in the video data is obtained; a difference between a timestamp of the first audio and a timestamp of the first video is less than a threshold.

6. The audio and video synchronization method according to claim 1, characterized in that: Also includes: The similarity between the first text and the second text is calculated by a text similarity algorithm.

7. An audio and video synchronization device, characterized in that: include: A data acquisition module, used to acquire a first audio and a first video, wherein the first video includes a lip image, and the lip image is used to represent the lip movement of the speaker; A video text acquisition module, used to determine a first text and a time of the first text based on a lip image in the first video; The first text is used to represent the content of the speaker's speech; An audio-text acquisition module, used to determine a second text and a time of the second text based on the first audio, wherein the second text is semantic information of the first audio; A similarity calculation module, used for calculating the similarity between the first text and the second text by using a text similarity algorithm; A time difference acquisition module, configured to determine a time difference when the similarity between the first text and the second text is higher than a preset similarity, wherein the time difference is a difference between the time of the first text and the time of the second text; A synchronization determination module, used to determine the magnitude relationship between the time difference and a preset duration; A synchronization adjustment module is used to adjust the starting time of the first audio or the starting time of the first video if the time difference is greater than or equal to the preset duration, so as to achieve synchronization between the first audio and the first video.

8. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the audio and video synchronization method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio and video synchronization method according to any one of claims 1 to 6 is implemented.

Images