Sound and picture synchronization method and device, electronic equipment and storage medium

Abstract

The invention relates to a sound and picture synchronization method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a subtitle file corresponding to a video file; obtaining an ith initial audio clip corresponding to the ith translated subtitle text, and determining subtitle display time and a subtitle character identifier corresponding to the ith translated subtitle text based on a subtitle time axis; processing the ith initial audio clip into a target audio clip matched with the subtitle display time corresponding to the ith translated subtitle text according to the clip duration of the ith initial audio clip, the subtitle display time corresponding to the ith translated subtitle text and the subtitle character identifier; and generating a target audio file synchronized with the video file according to the target audio clip matched with the subtitle display time of each translated subtitle text in the subtitle file. Therefore, the target audio file synchronized with the video file can be efficiently, quickly and automatically generated, the labor cost is reduced, and the audio and picture synchronization efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a method and device for synchronizing audio and video, an electronic device, and a storage medium. Background Art

[0002] In recent years, with the continuous improvement of the quality of domestic film and television dramas, many popular domestic film and television dramas have also appeared on foreign TV screens and overseas broadcasting platforms. In order to better export film and television dramas, the subtitles of film and television dramas can be translated into the language of the target market and dubbed. However, due to the differences in the length of subtitle texts and pronunciation systems (such as speaking speed) between different languages, simply dubbing the video directly can easily cause "overlapping sounds" or "asynchronous sound and picture". For example, for clips with dense lines, if the dubbing is based on the subtitle start time of the original drama subtitle timeline, if the current audio clip is too long, it is likely that the current audio clip has not yet finished and the next audio clip has begun. This overlap of the previous and next audio clips causes the "overlapping sound" phenomenon, which is also a phenomenon of "asynchronous sound and picture".

[0003] In order to solve the problem of audio and picture asynchrony in dubbing between different languages ​​of film and television dramas, it is usually necessary to manually adjust the audio duration and playback time, but this method has high labor costs and low efficiency. Summary of the invention

[0004] In view of this, the present disclosure proposes a method and device for audio and video synchronization, an electronic device and a storage medium, which can automatically generate a target audio file synchronized with a video file efficiently and quickly, achieve audio and video synchronization without human intervention, reduce labor costs, and improve audio and video synchronization efficiency.

[0005] According to one aspect of the present disclosure, a method for synchronizing audio and video is provided, comprising: obtaining a subtitle file corresponding to a video file, the subtitle file comprising a plurality of translated subtitle texts and a subtitle timeline; the translated subtitle text comprising a translation text of an original subtitle text of the video file into a target language; the subtitle timeline being a timeline adopted by the original subtitle text of the video file, the subtitle timeline representing a subtitle display time of the subtitle text and a subtitle character identifier, the subtitle character identifier representing a character who speaks the subtitle text, the subtitle display time comprising a start time and an end time of the subtitle text display; for an i-th translated subtitle text in the subtitle file, obtaining a subtitle time for the i-th translated subtitle text; The method comprises the steps of: obtaining an i-th initial audio segment corresponding to the video file and determining the subtitle display time and subtitle character identification corresponding to the i-th translated subtitle text based on the subtitle timeline; wherein the i-th initial audio segment is an audio segment generated based on the i-th translated subtitle text; processing the i-th initial audio segment into a target audio segment matching the subtitle display time corresponding to the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time and subtitle character identification corresponding to the i-th translated subtitle text; and generating a target audio file synchronized with the video file according to the target audio segments matching the subtitle display time of each translated subtitle text in the subtitle file.

[0006] According to another aspect of the present disclosure, another audio-visual synchronization method is provided, comprising: obtaining a subtitle file corresponding to a video file, the subtitle file comprising a plurality of translated subtitle texts and a subtitle timeline; the translated subtitle text comprising a translation text of an original subtitle text of the video file into a target language; the subtitle timeline being a timeline adopted by the original subtitle text of the video file, the subtitle timeline representing a subtitle display time and a subtitle character identifier of the subtitle text, the subtitle character identifier representing a character who speaks the subtitle text, the subtitle display time comprising a start time and an end time of the subtitle text display; in a case where the subtitle timeline represents that the subtitle density is less than or equal to a specified density threshold, based on the subtitle file, executing an audio-visual hard synchronization strategy, and obtaining a target audio file synchronized with the video file; the subtitle density represents the density of the subtitle text; in a case where the subtitle timeline represents that the subtitle density is greater than the specified density threshold, based on the subtitle file, executing an audio-visual soft synchronization strategy, and obtaining a target audio file synchronized with the video file.

[0007] According to another aspect of the present disclosure, a sound and picture synchronization device is provided, comprising: a first acquisition module, used to acquire a subtitle file corresponding to a video file, the subtitle file comprising a plurality of translated subtitle texts and a subtitle timeline; the translated subtitle text comprises a translation text of an original subtitle text of the video file into a target language; the subtitle timeline is a timeline adopted by the original subtitle text of the video file, the subtitle timeline represents a subtitle display time of the subtitle text and a subtitle character identifier, the subtitle character identifier represents a character who speaks the subtitle text, the subtitle display time comprises a start time and an end time of the subtitle text display; a second acquisition module, used to acquire, for an i-th translated subtitle text in the subtitle file, the i-th translated subtitle text The invention relates to an i-th initial audio segment corresponding to the video file, and determines the subtitle display time and subtitle character identification corresponding to the i-th translated subtitle text based on the subtitle timeline; wherein the i-th initial audio segment is an audio segment generated based on the i-th translated subtitle text; a processing module is used to process the i-th initial audio segment into a target audio segment matching the subtitle display time corresponding to the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time and subtitle character identification corresponding to the i-th translated subtitle text; a generating module is used to generate a target audio file synchronized with the video file according to the target audio segments matching the subtitle display time of each translated subtitle text in the subtitle file.

[0008] According to another aspect of the present disclosure, another audio-visual synchronization device is provided, comprising: a file acquisition module, used to acquire a subtitle file corresponding to a video file, the subtitle file comprising a plurality of translated subtitle texts and a subtitle timeline; the translated subtitle text comprising a translation text of an original subtitle text of the video file into a target language; the subtitle timeline is a timeline adopted by the original subtitle text of the video file, the subtitle timeline represents a subtitle display time of the subtitle text and a subtitle character identifier, the subtitle character identifier represents a character who speaks the subtitle text, the subtitle display time comprises a start time and an end time of the subtitle text display; a first execution module, used to, when the subtitle density represented by the subtitle timeline is less than or equal to a specified density threshold, execute an audio-visual hard synchronization strategy based on the subtitle file to obtain a target audio file synchronized with the video file; and a second execution module, used to, when the subtitle density represented by the subtitle timeline is greater than the specified density threshold, execute an audio-visual soft synchronization strategy based on the subtitle file to obtain a target audio file synchronized with the video file.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0010] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0012] According to various aspects of the present disclosure, by adopting the subtitle display time and subtitle character identification indicated by the subtitle timeline adopted by the original subtitle text of the video file, the initial audio segment corresponding to the translated subtitle text translated from the original subtitle text can be automatically processed into a target audio segment matching the subtitle display time, so that a target audio file synchronized with the video file can be efficiently and quickly generated, and the audio and video synchronization effect can be achieved without human intervention, greatly reducing labor costs.

[0013] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0015] Figure 1 A flow chart of a method for synchronizing audio and video according to an embodiment of the present disclosure is shown.

[0016] Figure 2 A schematic diagram of a subtitle timeline and subtitle text according to an embodiment of the present disclosure is shown.

[0017] Figure 3 A schematic diagram of an audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0018] Figure 4 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0019] Figure 5 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0020] Figure 6 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0021] Figure 7 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0022] Figure 8 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0023] Fig. 9 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0024] Fig.10 A flow chart of a hard audio-video synchronization strategy according to an embodiment of the present disclosure is shown.

[0025] Fig.11 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0026] Fig.12 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0027] Fig.13 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0028] Fig.14 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0029] Fig.15 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0030] Fig.16 A schematic diagram showing another audio and video synchronization logic according to an embodiment of the present disclosure is shown.

[0031] Fig.17 A flow chart of a soft audio-video synchronization strategy according to an embodiment of the present disclosure is shown.

[0032] Fig.18 A block diagram of a device for synchronizing audio and video according to an embodiment of the present disclosure is shown.

[0033] Fig.19 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0034] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0035] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0036] The term "and / or" herein is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent any one or more elements selected from the set consisting of A, B, and C. In the description of the present disclosure, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0037] It should be understood that the terms "first", "second", etc. in the claims, specifications and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0038] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0039] As mentioned above, in order to better carry out the export of domestic film and television dramas, the subtitles of the film and television dramas can be translated into the target language of the target country and dubbed. However, due to the differences in the length of the subtitle text and the pronunciation system (such as speech speed) between different languages, simply dubbing directly can easily cause the phenomenon of "overlapping" or "asynchronous sound and picture". In order to solve the problem of asynchronicity between sound and picture in the dubbing production of film and television dramas, that is, to solve the problem of asynchronicity between the original video screen and the audio and video of the dubbing audio in other languages, the embodiment of the present disclosure proposes a method for synchronizing sound and picture, which can be based on the segment length of the initial audio segment corresponding to the translated subtitle text and the subtitle time axis used by the original subtitle text of the video file, and realize the full automation of the initial audio segment without manual intervention. Acceleration, shifting, merging and other processing, to efficiently generate a target audio file synchronized with the video file, and can also combine the acceleration ratio, offset, number of merges and other parameters, to achieve a better audio and video synchronization effect, which is conducive to ensuring the deep integration and harmonious resonance of the audio and video of the video file and the target audio file, so as to bring a smooth audio-visual experience to the audience, and significantly reduce the labor cost of the dubbing-related business link.

[0040] The audio and video synchronization method of the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure may also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure may be a touch screen, a non-touch screen, or a screenless terminal device. The touch screen terminal device can be controlled by clicking and sliding on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, a keyboard, a touch panel, and the terminal device is controlled through the input device. For example, the device without a screen may be a Bluetooth speaker without a screen. For example, the terminal device of the present application may include but is not limited to a user equipment (UE), a mobile device, a user terminal, a terminal, a handheld device, a tablet computer, a laptop, a PDA, a computing device, etc.

[0041] The audio and video synchronization method of the embodiment of the present disclosure can also be deployed on a server. The server can be located in the cloud or locally. It can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function. The wireless connection function refers to the ability to connect to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiment of the present disclosure may also have the function of communicating through a wired connection. For example, the server of the embodiment of the present disclosure can be located in the cloud, communicate with the terminal device, receive the subtitle file of the video file sent by the terminal device, and use the audio and video synchronization method deployed on the server to generate a target audio file synchronized with the video file based on the subtitle file, and return it to the terminal device to generate the target audio file for the user in the terminal device.

[0042] Figure 1 FIG. 1 is a flow chart showing a method for synchronizing audio and video according to an embodiment of the present disclosure. The method can be applied to electronic devices such as the above-mentioned terminal device or server. Figure 1 As shown, the method includes: step S11 to step S14.

[0043] In step S11, a subtitle file corresponding to the video file is obtained;

[0044] Among them, the subtitle file includes multiple translated subtitle texts and a subtitle timeline; the translated subtitle text includes the original subtitle text of the video file translated into the target language; the subtitle timeline is the timeline used by the original subtitle text of the video file, the subtitle timeline represents the subtitle display time and subtitle character identification of the subtitle text, the subtitle character identification represents the character who speaks the subtitle text (that is, represents the speaker of the subtitle text, or represents the role of speaking the subtitle text), and the subtitle display time includes the start time and end time of the subtitle text display.

[0045] Wherein, the original subtitle text and the translated subtitle text of the video file are texts of different languages, for example, the original subtitle text is Chinese text, and the translated subtitle text is Thai text, English text, etc. Those skilled in the art can translate the original subtitle text corresponding to the video file into a translated subtitle text of any target language according to actual needs, and this embodiment of the present disclosure is not limited to this. It should be understood that the subtitle display time and the subtitle character identification of the translated subtitle text and the original subtitle text are consistent, that is, the subtitle time axis adopted by the original subtitle text is also the subtitle time axis required to be adopted by the translated subtitle text.

[0046] For example, Figure 2 A schematic diagram showing a subtitle timeline and subtitle text, such as Figure 2As shown, the subtitle time axis includes the subtitle display time (t0, t1, ..., t 13 ) and subtitle character identification (speaker1, speaker2, speaker3). For example, the start time of the first translated subtitle text "auspicious day" is t0, the end time is t1, and the corresponding subtitle character identification is "speaker1", and so on.

[0047] In step S12, for the ith translated subtitle text in the subtitle file, the ith initial audio segment corresponding to the ith translated subtitle text is obtained, and the subtitle display time and subtitle character identification corresponding to the ith translated subtitle text are determined based on the subtitle timeline; wherein the ith initial audio segment is an audio segment generated based on the ith translated subtitle text.

[0048] The ith initial audio segment corresponding to the ith translated subtitle text can be obtained in sequence according to the order of the subtitle texts indicated in the subtitle timeline, and the subtitle display time and subtitle character identification corresponding to the ith translated subtitle text can be determined. The "one" translated subtitle text refers to a translated subtitle text between a starting time and a corresponding ending time in the subtitle timeline, that is, a subtitle text displayed on the screen at the same time.

[0049] In actual applications, the initial audio segment corresponding to each translated subtitle text can be obtained in advance through manual dubbing, or the audio synthesis technology known in the art can be used to synthesize the translated subtitle text into the corresponding initial audio segment based on the audio features such as the tone and timbre of the characters' speech in the original audio segment of the original subtitle text. The embodiments of the present disclosure are not limited to this.

[0050] As mentioned above, the subtitle timeline used by the original subtitle text is also the subtitle timeline required to be used by the translated subtitle text. Therefore, based on the subtitle timeline, the subtitle display time and subtitle character identification of the original subtitle text corresponding to the i-th translated subtitle text can be determined, that is, the subtitle display time and subtitle character identification corresponding to the i-th translated subtitle text can be determined.

[0051] In step S13, the i-th initial audio segment is processed into a target audio segment that matches the subtitle display time corresponding to the i-th translated subtitle text according to the segment length of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identification.

[0052] It should be understood that the subtitle display time of the subtitle text on the subtitle time axis is equivalent to the playback time of the dubbing audio segment corresponding to the subtitle text, or in other words, the display time of the subtitle text should be aligned with the playback time of the dubbing audio segment corresponding to the subtitle text, so as to achieve the audio-visual synchronization effect when playing the video. However, due to the differences in the length of the subtitle text and the pronunciation system (such as speech speed) between different languages, the initial audio segment corresponding to the translated subtitle text may not be exactly aligned with the subtitle display time indicated by the original subtitle time axis of the video file. For example, the segment length of the initial audio segment may be less than the subtitle display time indicated on the subtitle time axis, or it may be greater than the subtitle display time indicated on the subtitle time axis. If the playback of the initial audio segment corresponding to the translated subtitle text is directly controlled according to the original subtitle time axis, the phenomenon of audio-visual asynchrony may occur. Therefore, the segment length of the initial audio segment corresponding to each translated subtitle text, the subtitle display time corresponding to each translated subtitle text, and the subtitle character identification can be used to process the initial audio segment corresponding to each translated subtitle text into a target audio segment matching the subtitle display time corresponding to each translated subtitle text, so as to generate a target audio file synchronized with the video file.

[0053] Specifically, in a possible implementation, the above step S13, according to the segment length of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier, processing the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text, may include:

[0054] Step S1300, when the segment duration of the i-th initial audio segment is less than or equal to the subtitle display duration of the i-th translated subtitle text, determining the i-th initial audio segment as a target audio segment that matches the subtitle display duration of the i-th translated subtitle text;

[0055] Step S1301, when the segment duration of the i-th initial audio segment is greater than the subtitle display duration of the i-th translated subtitle text, a ratio between the segment duration of the i-th initial audio segment and the subtitle display duration of the i-th translated subtitle text is determined as a speedup ratio corresponding to the i-th initial audio segment;

[0056] Step S1302, when the acceleration ratio corresponding to the ith initial audio segment is less than or equal to a preset acceleration ratio threshold, the ith initial audio segment is accelerated according to the acceleration ratio corresponding to the ith initial audio segment to obtain a target audio segment that matches the subtitle display time of the ith translated subtitle text.

[0057] Among them, the subtitle display duration of the ith translated subtitle text is determined based on the subtitle display time of the ith translated subtitle text. It should be understood that the subtitle display duration of the translated subtitle text (i.e., the duration between the end time and the start time) can be obtained by knowing the start time and end time of the translated subtitle text display. Those skilled in the art can customize the acceleration ratio threshold according to actual needs. Setting the acceleration ratio threshold is equivalent to setting a maximum acceleration threshold to prevent the accelerated audio speed from being too fast and affecting the listening experience.

[0058] Considering that, in the above step S1300, the i-th initial audio segment is directly determined as the target audio segment that matches the subtitle display time of the i-th translated subtitle text, there may be a blank period after the target audio segment corresponding to the i-th translated subtitle text and before the start time of display of the i+1-th translated subtitle text. Therefore, the target audio segment can be pushed into the audio queue, and the blank audio segment is used to fill the blank after the target audio segment to ensure that the subsequently generated target audio segment is aligned with the subtitle display time indicated on the subtitle timeline; wherein, the audio queue is used to store the target audio segment and the blank audio segment in sequence, and the blank audio segment is an audio segment without sound.

[0059] Therefore, after obtaining the target audio segment that matches the subtitle display time of the i-th translated subtitle text, the method may further include: step S1303, pushing the target audio segment to the audio queue, and based on the first deviation between the segment cutoff time when the target audio segment is aligned to the subtitle timeline (that is, after aligning the start time of the target audio segment with the start time of the i-th translated subtitle text) and the start time of display of the i+1-th translated subtitle text on the subtitle timeline, filling a blank audio segment corresponding to the first deviation after the target audio segment in the audio queue.

[0060] For example, Figure 3 As shown in FIG. 1 , the first initial audio segment (P1) of the first English subtitle text “auspicious day” (i.e., the first translated subtitle text) corresponding to the first original subtitle text “auspicious day” is shorter than the subtitle display duration of the first English subtitle text (i.e., the duration between t0 and t1). Then, P1 can be added to the audio queue as the target audio segment M1 of the first English subtitle text. Then, M1 can be aligned to the subtitle timeline (i.e., the start time of M1 is aligned with the start time of the first English subtitle text) according to the segment cutoff time t M1 The first deviation (i.e., t2-t M1), fill the blank audio segment (K1) after M1 in the audio queue, where the segment length of K1 is t2-t M1 Length of time.

[0061] For example, Figure 4 As shown in the figure, the segment duration of the second initial audio segment (P2) corresponding to the second translated subtitle text "young lady" corresponding to the second original subtitle text "girl" is greater than the subtitle display duration of the second translated subtitle text (i.e., the duration between t2 and t3). At this time, the ratio between the segment duration of P2 and the subtitle display duration (i.e., the duration between t2 and t3) is calculated as the acceleration ratio corresponding to P2 (e.g., the acceleration ratio is 1.1). If the acceleration ratio corresponding to P2 (e.g., 1.1) is less than the preset acceleration ratio threshold (e.g., the acceleration ratio threshold is 1.25), then P1 is accelerated according to the acceleration ratio corresponding to P2 (e.g., 1.1) to obtain the target audio segment M2 that matches the subtitle display time (i.e., t2 and t3), that is, P1 is accelerated to the target audio segment M2 at 1.1 times the speed, and then M2 can be pushed to the audio queue and aligned to the segment cutoff time t on the subtitle timeline according to M2. M2 (At this time t M2 =t3) and the start time t4 of the third translated subtitle text "It's time to get married" on the subtitle timeline (i.e., t4-t M2 ), fill the blank audio segment (K2) after M2 in the audio queue, where the segment length of K2 is t2-t M2 It should be understood that if M2 is aligned to the end time t of the segment on the subtitle timeline M2 The first deviation t4-t from the start time t4 of the third translated subtitle text display on the subtitle timeline M2 If it is 0, that is, there is no gap between the end time of the second translated subtitle text and the start time of the third translated subtitle text, then there is no need to fill the blank audio segment after M2 in the audio queue.

[0062] In practical applications, after the initial audio clip is accelerated according to the acceleration ratio, the acceleration ratio can be marked on the accelerated target audio clip, so that in some scenarios where the original speed needs to be restored, the marked acceleration ratio can be used to restore the target audio clip to the original speed of the initial audio clip.

[0063] Among them, it is also possible that the acceleration ratio corresponding to the initial audio segment is greater than the preset acceleration ratio threshold. In this case, if the acceleration is performed according to the acceleration ratio, it may result in a large difference from the original speech speed, affecting the listening experience. Therefore, in a possible implementation, the above step S13, based on the segment length of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier, processes the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text, which may include:

[0064] Step S1304: when the acceleration ratio corresponding to the ith initial audio segment is greater than the acceleration ratio threshold, based on the subtitle display time of the ith translated subtitle text, determine whether there are sufficient blank segments before and after the ith initial audio segment; wherein the existence of sufficient blank segments includes that there are blank segments before and / or after the ith initial audio segment and the duration of the blank segments is greater than or equal to the acceleration offset corresponding to the ith initial audio segment;

[0065] The acceleration offset includes the difference between the duration of the audio segment obtained by accelerating the ith initial audio segment using the acceleration ratio threshold and the subtitle display duration corresponding to the ith translated subtitle text; the duration of the blank segment before the ith initial audio segment includes the duration of the blank audio segment filled before the ith initial audio segment; the duration of the blank segment after the ith initial audio segment includes the duration from the end time of displaying the ith translated subtitle text to the start time of displaying the i+1th translated subtitle text on the subtitle timeline;

[0066] Step S1305: If there are sufficient blank segments before and after the ith initial audio segment, the ith initial audio segment is accelerated according to the acceleration ratio threshold to obtain a target audio segment that matches the subtitle display time of the ith translated subtitle text;

[0067] Step S1306: Push the target audio segment into the audio queue by occupying a blank segment before the i-th initial audio segment and / or occupying a blank segment after the i-th initial audio segment.

[0068] In actual applications, after the i-th initial audio clip is accelerated according to the acceleration ratio threshold to obtain the target audio clip, the blank audio clip filled before the i-th initial audio clip in the audio queue can be compressed according to the above-mentioned acceleration offset, or the blank space between the end time of the i-th translated subtitle text display and the start time of the i+1-th translated subtitle text display on the subtitle timeline can be directly occupied until the above-mentioned acceleration offset requirement is met, and the target audio clip accelerated according to the acceleration ratio threshold can be pushed into the audio queue.

[0069] In practical applications, any occupation method in the art can be used in the above step S1306 to occupy the blank segments before and / or after the i-th initial audio segment, as long as the target audio segment pushed into the audio queue does not overlap with the target audio segments before and after it. This is not limited in the embodiments of the present disclosure.

[0070] Considering that directly compressing the blank audio segment before or occupying the blank after based on the acceleration offset may make the target audio segment too early or late relative to the subtitle display time, an offset threshold may be set to limit the compression amount or occupation amount of the blank segment. Specifically, the above step S1306, pushing the target audio segment to the audio queue by occupying the blank segment before the i-th initial audio segment and / or occupying the blank segment after the i-th initial audio segment, may include:

[0071] Step S13061, when there are sufficient blank segments before and after the ith initial audio segment, based on a preset offset threshold, an acceleration offset, and the duration of the blank segment before the ith initial audio segment, determine the required compression amount of the blank segment before the ith initial audio segment and the amount of the blank segment after the ith initial audio segment occupied by the target audio segment; wherein the offset threshold is used to limit the maximum compression amount of the blank segment;

[0072] Step S13062: When the occupied amount is less than or equal to the duration of the blank segment after the i-th initial audio segment, based on the compression amount, compress the blank audio segment filled after the target audio segment corresponding to the i-1-th translated subtitle text in the audio queue, and push the target audio segment matching the subtitle display time of the i-th translated subtitle text into the audio queue and arrange it after the compressed blank audio segment;

[0073] Step S13063, based on the second deviation between the segment cutoff time when the target audio segment corresponding to the i-th translated subtitle text in the audio queue is aligned to the subtitle timeline (that is, the target audio segment is arranged after the compressed blank audio segment) and the start time when the i+1-th translated subtitle text is displayed on the subtitle timeline, a blank audio segment corresponding to the second deviation is filled after the target audio segment corresponding to the i-th translated subtitle text in the audio queue.

[0074] For example, Figure 5As shown in FIG. 1 , the segment duration of the third initial audio segment (P3) corresponding to the third translated subtitle text “It's time to get married” corresponding to the third original subtitle text “It's time to get married” is greater than the subtitle display duration of the third translated subtitle text (i.e., the duration between t4 and t5). At this time, the ratio between the segment duration of P3 and the subtitle display duration (i.e., the duration between t4 and t5) is calculated as the acceleration ratio corresponding to P3. If the acceleration ratio corresponding to P3 (e.g., 1.6) is greater than a preset acceleration ratio threshold (e.g., the acceleration ratio threshold is 1.25), and if the duration of the blank segment before P3 (i.e., the blank audio segment K2) (the duration between t4 and t3) is greater than the duration between t4 and t5), then the acceleration ratio corresponding to P3 is calculated. 34 ) plus the duration of the blank segment after P3 (i.e., the duration T between the end time t5 of the third translated subtitle text and the start time t6 of the fourth translated subtitle text). 56 ) is greater than or equal to the difference between the duration of the audio segment obtained by accelerating P3 using the acceleration ratio threshold of 1.25 and the subtitle display duration of the third translated subtitle text (i.e., the duration between t4 and t5), for example, T 34 +T 56 The length of the audio clip obtained by accelerating P3 using the acceleration ratio threshold of 1.25 is 12.8 seconds. The subtitle display duration of the third translated subtitle text (i.e., the duration between t4 and t5) is 10 seconds. Since the acceleration offset (i.e., the difference between 12.8 seconds and 10 seconds is 2.8 seconds) is less than the above T 34 +T 56 If the length of the audio clip obtained by accelerating P3 by using the acceleration ratio threshold of 1.25 is 15 seconds, then the difference of 5 seconds between 15 seconds and 10 seconds is greater than the above T 34 +T 56 4 seconds, it is considered that there is no sufficient blank segment before and after P3;

[0075] Then, if there are sufficient blank segments before and after P3, P3 can be accelerated using the acceleration ratio threshold of 1.25 to obtain the target audio segment M3 corresponding to the third translated subtitle text; at this time, since the segment duration of the target audio segment M3 obtained by accelerating using the acceleration ratio threshold of 1.25 is greater than the corresponding subtitle display duration (i.e., the duration between t4 and t5), if M3 is directly pushed to the audio queue, a large amount of blank space between t5 and t6 will be occupied. Therefore, if there is a blank audio segment (such as K2) before P3, the blank audio segment (such as K2) before P3 can be compressed, that is, the blank audio segment can be shortened. The duration of the blank audio segment (such as K2) and the compression amount (i.e., the shortened duration) of the blank audio segment (such as K2) can be determined according to a preset offset threshold, the acceleration offset of the target audio segment M3, and the duration of the blank audio segment (such as K2). Specifically, when the duration of the blank audio segment is greater than or equal to the offset threshold and the acceleration offset is less than or equal to the offset threshold, the compression amount of the blank audio segment (such as K2) is the acceleration offset; when the duration of the blank audio segment is greater than or equal to the offset threshold and the acceleration offset is greater than the offset threshold, the compression amount of the blank audio segment is the offset threshold; when the duration of the blank audio segment is less than the offset threshold And when the acceleration offset is greater than the duration of the blank audio segment, the compression amount of the blank audio segment is the duration of the blank audio segment (that is, the blank audio segment is removed); when the duration of the blank audio segment is less than the offset threshold and the acceleration offset is less than the duration of the blank audio segment, the compression amount of the blank audio segment is the acceleration offset; after determining the compression amount of the blank audio segment (such as K2), the difference between the acceleration offset and the compression amount can be used as the occupancy of the blank segment after M3 to P3, and when the occupancy of the blank segment after P3 by M3 is less than or equal to the duration of the blank segment after P3, the compression can be executed. M3 is pushed to the audio queue for processing. For example, if the acceleration offset is 2.8 seconds, the offset threshold is 2 seconds, and the duration of K2 is 3 seconds, then the compression amount of K2 is the maximum compression amount of 2 seconds limited by the offset threshold, which is equivalent to shortening the duration of K2 by 2 seconds. Then the shortened duration of K2 is 1 second. At this time, the difference between the acceleration offset of 2.8 seconds and the compression amount of 2 is 0.8 seconds, which is the occupation of the blank segment after P3 by M3. If the occupation amount of 0.8 seconds is less than or equal to the duration between t5 and t6 (equivalent to that under the limitation of the above offset threshold, there is still enough blank space between t5 and t6 for the target audio segment to occupy), then in this case, Figure 5 As shown, M3 can be pushed to the audio queue. M3 pushed to the audio queue will occupy the blank space of 0.8 seconds after t5, that is, M3 pushed to the audio queue corresponds to the segment cutoff time t on the subtitle timeline. M3There is a difference of 0.8 seconds between t5 and M3, and the starting time of M3 is 2 seconds before t4, that is, it occupies the 2 seconds of blank space before t4; if the acceleration offset is 1.8 seconds, the offset threshold is 2 seconds, and the duration of K2 is 3 seconds, then the compression amount of K2 is the acceleration offset of 1.8 seconds (that is, K2 only needs to be compressed by 1.8 seconds), which can meet the duration requirement of M3. At this time, it is equivalent to shortening the duration of K2 by 1.8 seconds. Then the shortened duration of K2 is 1.2 seconds. In this case, the occupation of the blank segment after P3 is 0. In this case, Figure 6 As shown, M3 can be pushed into the audio queue, and the M3 pushed into the audio queue corresponds to the segment cutoff time t on the subtitle timeline. M3 =t5, the starting time is 1.8 seconds before t4; if the acceleration offset is 2.8 seconds, the offset threshold is 2 seconds, and the duration of K2 is 1.5 seconds, then the compression amount of K2 is 1.5 seconds of the duration of K2, which is equivalent to shortening the duration of K2 by 1.5 seconds, that is, removing K2. In this case, the duration of the blank segment after P3 is greater than or equal to the occupancy of the blank segment after P3, which is 1.3 seconds. In this case, Figure 7 As shown, M3 can be pushed to the audio queue. M3 pushed to the audio queue will occupy the blank space of 1.3 seconds after t5, that is, M3 pushed to the audio queue corresponds to the segment cutoff time t on the subtitle timeline. M3 The difference between t5 and the start time is t3, and the maximum compression amount of the blank segment is limited by the offset threshold, so as to avoid the target audio segment being too far forward or backward relative to the corresponding subtitle display time.

[0076] Then, after pushing M3 to the audio queue, Figure 7 As shown, the segment cutoff time t can be aligned to the subtitle timeline according to M3 M3 The second deviation (i.e., t6-t M3 ), a blank audio segment (K3) is filled after M3 in the audio queue, wherein the segment length of the blank audio segment (K3) is t6-t M3 It should be understood that if M3 is aligned to the end time of the segment on the subtitle timeline, M3 The second deviation t6-t from the start time t6 of the display of the fourth translated subtitle text on the subtitle timeline M3 If it is 0, no blank audio segment will be filled after M3 in the audio queue.

[0077] It should be understood that the implementation method of the above steps S13061 to S13063 is equivalent to preferentially moving the target audio segment forward when there are sufficient blank segments before and after the initial audio segment, that is, preferentially compressing the blank segments before the initial audio segment, and then occupying the blank segments after the initial audio segment; in actual applications, the target audio segment can also be preferentially moved backward based on the offset threshold, that is, preferentially occupying the blank segments after the initial audio segment, that is, first occupying as many blank segments after the initial audio segment as possible within the allowed range, and then compressing the blank segments before the initial audio segment as needed, and the embodiments of the present disclosure do not limit this.

[0078] It should be understood that if there are sufficient blank segments only after the i-th initial audio segment, that is, there are no blank segments before the i-th initial audio segment, the target audio segment corresponding to the i-th translated subtitle text can be directly pushed into the audio queue. At this time, the target audio segment corresponding to the i-th translated subtitle text is arranged after the target audio segment corresponding to the i-1-th translated subtitle text; of course, it is also possible to limit the target audio segment's occupancy of the blank segment after the initial audio segment to not exceed an offset threshold. If the occupancy exceeds the offset threshold, reference can be made to the processing method for when there are no sufficient blank segments before and after the initial audio segment in the following text to obtain a target audio segment that meets the requirements.

[0079] Among them, it is also possible that the target audio segment occupies a larger amount of the blank segment after the ith initial audio segment than the duration of the blank segment after the ith initial audio segment. For example, if the acceleration offset is 2.8 seconds, the offset threshold is 2 seconds, and the duration of K2 is 3 seconds, then the compression amount of K2 is the maximum compression amount of 2 seconds limited by the offset threshold. The difference between the acceleration offset of 2.8 seconds and the compression amount of 2, 0.8 seconds, is the occupancy of the blank segment after P3. If the duration of the blank segment after P3 (the duration between t5 and t6) is 0.7 seconds, it means that the occupancy of 0.8 seconds is greater than the duration between t5 and t6. In other words, although the duration of K2 is 3 seconds, The sum of the duration between t5 and t6 (0.7 seconds) (3.7 seconds) is greater than the accelerated offset of the initial audio segment (2.8 seconds). However, under the above-mentioned offset threshold, there is not enough blank space between t5 and t6 for the target audio segment M3 to occupy. In this case, the processing method described later when there are not enough blank segments before and after the i-th initial audio segment can be used to obtain a target audio segment that meets the requirements. That is, the situation where the target audio segment occupies the blank segment after the i-th initial audio segment longer than the duration of the blank segment after the i-th initial audio segment can also be considered as a situation where there are not enough blank segments before and after the i-th initial audio segment.

[0080] In a possible implementation, the above step S13, processing the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier, may include:

[0081] Step S1307: when the acceleration ratio corresponding to the ith initial audio segment is greater than the acceleration ratio threshold and there are no sufficient blank segments before and after the ith initial audio segment, based on the subtitle timeline, determine whether the ith translated subtitle text and the (i+1)th translated subtitle text correspond to the same subtitle character identifier; wherein, the absence of sufficient blank segments includes that there are no blank segments before and after the ith initial audio segment, or the duration of the existing blank segments is less than the acceleration offset corresponding to the ith initial audio segment, or the amount of the blank segment after the ith initial audio segment occupied by the target audio segment is greater than the duration of the blank segment after the ith initial audio segment;

[0082] Step S1308, when the i-th translated subtitle text and the i+1-th translated subtitle text correspond to the same subtitle character identifier, merging the i-th initial audio segment corresponding to the i-th translated subtitle text and the i+1-th initial audio segment corresponding to the i+1-th translated subtitle text to obtain a merged audio segment;

[0083] Step S1309, based on the segment length of the merged audio segment, the merged display time corresponding to the ith translated subtitle text and the (i+1)th translated subtitle text, and the subtitle character identification, the merged audio segment is processed into a target audio segment that matches the merged display time; wherein the merged display time includes the start time of the display of the ith translated subtitle text to the end time of the display of the (i+1)th translated subtitle text.

[0084] In step S1307, it is determined whether the i-th translated subtitle text and the i+1-th translated subtitle text correspond to the same subtitle character identifier, that is, it is determined whether the i-th translated subtitle text and the i+1-th translated subtitle text are spoken by the same character; in step S1308, the two initial audio clips are merged, that is, the two initial audio clips are spliced. The embodiment of the present disclosure does not limit the merging method of the audio clips; in step S1309, the merged audio clip can be used as a new i-th initial audio clip, and with reference to the processing process of the above steps S1300 to S1309, the merged audio clip is processed into a target audio clip that matches the merged display time according to the clip length of the merged audio clip, the merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier, which will not be elaborated here.

[0085] In practical applications, after obtaining the target audio segment that matches the combined display time, the method further includes: Step S1310, pushing the target audio segment that matches the combined display time into the audio queue, and based on the third deviation between the segment cut-off moment after aligning the target audio segment that matches the combined display time to the caption time axis (i.e., after the start moment of the target audio segment is aligned with the start moment of the i-th translated caption text) and the start moment of the display of the (i + 2)-th translated caption text on the caption time axis, filling in blank audio segments corresponding to the third deviation after the target audio segment that matches the combined display time in the audio queue.

[0086] Exemplarily, as Figure 8 shown, if the acceleration ratio of the 4th initial audio segment P4 corresponding to the 4th translated caption text "lady, be careful" of the 4th original caption text "girl, be careful" is greater than the acceleration ratio threshold and there are no sufficient blank segments before and after the 4th initial audio segment P4, based on the caption time axis, it can be determined that the 4th translated caption text and the 5th translated caption text "You should get on the sedan" correspond to the same caption character identifier "speaker1", then the 4th initial audio segment corresponding to the 4th translated caption text can be merged with the 5th initial audio segment P5 corresponding to the 5th translated caption text to obtain a merged audio segment P45; then, this merged audio segment P45 can be used as the new 4th initial audio segment, and referring to the processing procedures of the above steps S1300 to S1309, according to the segment duration of this merged audio segment P45, the combined display time corresponding to the 4th translated caption text and the 5th translated caption text (i.e., from t6 to t9), and the caption character identifier, the merged audio segment P45 is processed into a target audio segment M45 that matches the combined display time (i.e., from t6 to t9); then, this M45 can be pushed into the audio queue, and based on the segment cut-off moment t M45 (such as t M45 exactly equal to t9) and the start moment t 10 of the display of the 6th translated caption text on the caption time axis 10 The third deviation t M45 -t 10 -t M45 is filled with blank audio segments (K4) corresponding to the third deviation after the target audio segment M45 that matches the combined display time in the audio queue, where the segment duration of K4 is also the duration of t

[0087] In practical applications, the maximum number of merged audio segments (that is, the maximum number of merged initial audio segments that can be merged) can be set. For example, if the maximum number of merges is set to 2, and the number of merged initial audio segments in the merged audio segment has reached the maximum number of merges, then when the acceleration ratio of the merged audio segment obtained by merging the i-th initial audio segment with the i+1-th initial audio segment is greater than the acceleration ratio threshold and there are no sufficient blank segments before and after the merged audio segment, it is no longer determined whether the subtitle character identifier of the i+1-th translated subtitle text corresponding to the merged audio segment is the same as the subtitle character identifier of the i+2-th translated subtitle text, and refer to the following when the i-th initial audio segment is merged. The merged audio segment is processed into a target audio segment that matches the merged display time by a processing flow that there are no sufficient blank segments before and after the initial audio segment and the i-th translated subtitle text and the i+1-th translated subtitle text correspond to different subtitle character identifiers, that is, the merged audio segment can be directly accelerated according to the acceleration ratio corresponding to the merged audio segment to obtain the target audio segment that matches the merged display time, or the text content of the i-th translated subtitle text and the i+1-th translated subtitle text can be shortened to obtain a shortened merged audio segment, and the shortened merged audio segment is used as the new i-th initial audio segment to re-execute the above-mentioned processing process from step S1300 to step S1306. The text content of the i-th translated subtitle text and the i+1-th translated subtitle text can be shortened based on the model and algorithm that automatically reduces the number of words and does not change the semantics in the related art. If the number of initial audio segments merged in the merged audio segment has not reached the maximum number of merges, then when the acceleration ratio of the merged audio segment obtained by merging the i-th initial audio segment with the i+1-th initial audio segment is greater than the acceleration ratio threshold and there are no sufficient blank segments before and after the merged audio segment, the next initial audio segment can be merged until the maximum number of merges is reached or the next initial audio segment and the i-th initial audio segment do not have the same subtitle character identifier, and the merged audio segment is used as the new i-th initial audio segment to re-execute the processing process from step S1300 to step S1306.

[0088] In a possible implementation, the above step S13, processing the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier, may include:

[0089] Step S1311, when the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after the i-th initial audio segment, and the subtitle character identifiers corresponding to the i-th translated subtitle text and the (i + 1)-th translated subtitle text are different, accelerate the i-th initial audio segment according to the acceleration ratio corresponding to the i-th initial audio segment to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text;

[0090] Step S1312, push the target audio segment corresponding to the i-th translated subtitle text into the audio queue, and based on the fourth deviation between the segment cut-off moment after aligning the target audio segment corresponding to the i-th translated subtitle text to the subtitle timeline (i.e., after the start moment of the target audio segment is aligned with the start moment of the i-th translated subtitle text) and the start moment of the (i + 1)-th translated subtitle text displayed on the subtitle timeline, fill in blank audio segments corresponding to the fourth deviation after the target audio segment corresponding to the i-th translated subtitle text in the audio queue. Exemplarily, as Fig. 9 shown, for the 6th initial audio segment P6 corresponding to the 6th original subtitle text "Lucky stars shine brightly and auspicious times arrive" and the 6th English subtitle text "Lucky stars shine brightly and auspicious times arrive", if the acceleration ratio corresponding to this P6 is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after this P6, and the subtitle cultural relic identifier corresponding to the 6th translated subtitle text is "speaker2", which is different from the subtitle character identifier "speaker3" corresponding to the 7th translated subtitle text "Lift the sedan", then directly accelerate the 6th initial audio segment P6 according to the acceleration ratio corresponding to the 6th initial audio segment P6. For example, if the segment duration of P6 is 5 seconds and the subtitle display duration corresponding to the 6th original subtitle text is 2 seconds, then the acceleration ratio is 2.5. Then, P6 can be accelerated at 2.5 times speed to obtain a target audio segment M6 that matches the subtitle display time (i.e., t 10 and t 11 ). That is, P6 is accelerated into a target audio segment M6 at 2.5 times speed. Then, M6 can be added to the audio queue, and according to M6 aligned to the subtitle timeline (i.e., aligning the start moment of M6 to the start moment t 10 ) of the 6th original subtitle text, the segment cut-off moment t M6 (at this time t M6 = t 11 ) and the start moment t 12 of the 7th translated subtitle text displayed on the subtitle timeline, the fourth deviation (i.e., t 12 - t M6), fill the blank audio segment (K5) after M6 in the audio queue, where the segment length of K5 is t 12 -t M6 This method can be understood as, when the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after the i-th initial audio segment, and the i-th translated subtitle text and the i+1-th translated subtitle text correspond to different subtitle character identifiers, ignoring the acceleration ratio threshold, directly accelerating the i-th initial audio segment according to the acceleration ratio corresponding to the i-th initial audio segment, to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text.

[0091] Taking into account the situation that the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after the i-th initial audio segment, and the i-th translated subtitle text and the i+1-th translated subtitle text correspond to different subtitle character identifiers, the acceleration ratio threshold is directly ignored, and the target audio segment generated by accelerating the i-th initial audio segment according to the acceleration ratio corresponding to the i-th initial audio segment may be too fast, affecting the user's listening experience. Therefore, in a possible implementation, the above step S13, according to the segment length of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier, processes the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text, which may include:

[0092] Step S1313, when the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after the i-th initial audio segment, and the i-th translated subtitle text and the (i+1)-th translated subtitle text correspond to different subtitle character identifiers, shorten the text content of the i-th translated subtitle text, and obtain a shortened audio segment corresponding to the shortened i-th translated subtitle text;

[0093] Step S1314, processing the shortened audio segment into a target audio segment that matches the shortened subtitle display time of the i-th translated subtitle text according to the segment duration of the shortened audio segment, the shortened subtitle display time of the i-th translated subtitle text, and the subtitle character identifier;

[0094] Step S1315, push the target audio segment corresponding to the shortened i-th translated subtitle text into the audio queue, and based on the fifth deviation between the segment cutoff time after the target audio segment corresponding to the shortened i-th translated subtitle text is aligned to the subtitle timeline (that is, after the start time of the target audio segment is aligned with the start time of the shortened i-th translated subtitle text) and the start time of the display of the i+1-th translated subtitle text on the subtitle timeline, fill in the blank audio segment corresponding to the fifth deviation after the target audio segment corresponding to the shortened i-th translated subtitle text in the audio queue.

[0095] In step S1313, shortening the text content of the ith translated subtitle text may be, for example, removing uncritical words in the ith translated subtitle text, or replacing longer words in the ith translated subtitle text with shorter words, etc. As long as the text content of the ith translated subtitle text can be shortened without changing the original semantics, the embodiments of the present disclosure are not limited to this.

[0096] In step S1313, the shortened audio segment corresponding to the i-th translated subtitle text can be obtained by referring to the method for obtaining the initial audio segment. Specifically, it can be obtained by manual dubbing, or it can also be obtained by using audio synthesis technology known in the art, based on the audio features such as the tone and timbre of the characters' speech in the original audio segment of the original subtitle text, to synthesize the shortened translated subtitle text into the corresponding shortened audio segment. The embodiments of the present disclosure are not limited to this.

[0097] In step S1314, the shortened audio segment can be used as the new i-th initial audio segment, and referring to the processing process from step S1300 to step S1312, the shortened audio segment can be processed into a target audio segment that matches the shortened subtitle display time of the i-th translated subtitle text according to the segment duration of the shortened audio segment, the shortened subtitle display time of the i-th translated subtitle text, and the subtitle character identification, which will not be described in detail here. Among them, the shortened subtitle display time of the i-th translated subtitle text is still the subtitle display time of the i-th original subtitle text specified on the original subtitle timeline.

[0098] In step S1315, the implementation method of the above-mentioned step S1304 can be referred to to push the target audio segment corresponding to the shortened i-th translated subtitle text into the audio queue, and based on the alignment of the target audio segment corresponding to the shortened i-th translated subtitle text to the fifth deviation between the segment cutoff time on the subtitle timeline and the start time of display of the i+1-th translated subtitle text on the subtitle timeline, a blank audio segment corresponding to the fifth deviation is filled after the shortened target audio segment corresponding to the i-th translated subtitle text in the audio queue, which will not be elaborated here.

[0099] Optionally, based on the above steps S1300 to S1312, the present embodiment provides Fig.10 FIG. 1 is a flow chart of a hard synchronization strategy for audio and video. Fig.10 As shown, the audio and video hard synchronization strategy includes:

[0100] Frequency segment Pi;

[0101] Step S902, determining whether the segment duration of the i-th initial audio segment Pi is less than or equal to the subtitle display duration Ti of the i-th translated subtitle text; if so, proceeding to step S903; if not, proceeding to step S904;

[0102] Step S903, determining the i-th initial audio segment Pi as the target audio segment Mi that matches the subtitle display time of the i-th translated subtitle text, and proceeding to step S906;

[0103] Step S904, determining whether the acceleration ratio corresponding to the i-th initial audio segment Pi is less than or equal to a preset acceleration ratio threshold; if so, proceeding to step S905; if not, proceeding to step S907;

[0104] Step S905, accelerating the ith initial audio segment Pi according to the acceleration ratio corresponding to the ith initial audio segment Pi, obtaining a target audio segment Mi that matches the subtitle display time of the ith translated subtitle text, and proceeding to step S906;

[0105] Step S906, pushing the target audio segment Mi into the audio queue, and filling a blank audio segment after Mi;

[0106] Step S907, determine whether there are sufficient blank segments before and after the i-th initial audio segment Pi; if so, proceed to step S908; if not, proceed to step S912;

[0107] Step S908, based on a preset offset threshold, an acceleration offset, and a duration of a blank segment before the ith initial audio segment, determining a required compression amount of the blank segment before the ith initial audio segment Pi and an occupation amount of the blank segment after the ith initial audio segment Pi by the target audio segment Mi;

[0108] Step S909, determining whether the amount of the blank segment after the i-th initial audio segment occupied by the target audio segment is less than or equal to the duration Ti0 of the blank segment after the i-th initial audio segment; if so, proceeding to step S910; if not, proceeding to step S912;

[0109] Step S910, compressing the blank audio segment Ki-1 filled before the i-th initial audio segment Pi in the audio queue based on the compression amount required for the blank segment;

[0110] Step S911, accelerating the i-th initial audio segment Pi according to the acceleration ratio threshold, obtaining a target audio segment Mi that matches the subtitle display time of the i-th translated subtitle text, and proceeding to step S906;

[0111] Step S912, determining whether the i-th translated subtitle text Zi and the i+1-th translated subtitle text Zi+1 correspond to the same subtitle character identifier; if so, proceeding to step S913; if not, proceeding to step S914;

[0112] Step S913, merging the i-th initial audio segment Pi corresponding to the i-th translated subtitle text with the i+1-th initial audio segment Pi+1 corresponding to the i+1-th translated subtitle text to obtain a merged audio segment; then, the merged audio segment can be used as a new i-th initial audio segment to return to step S902 and re-execute the above audio-visual hard synchronization process;

[0113] Step S914: accelerate the ith initial audio segment Pi according to the acceleration ratio corresponding to the ith initial audio segment Pi to obtain a target audio segment Mi that matches the subtitle display time of the ith translated subtitle text, and proceed to step S906.

[0114] The above-mentioned audio and video hard synchronization process can allow the initial audio segment corresponding to each translated subtitle text to be slightly accelerated, merged or shifted forward or backward (i.e., compressed or occupied blank segments) relative to the subtitle display time indicated by the subtitle timeline, thereby ensuring that each translated subtitle text and the target audio segment can basically achieve audio and video hard synchronization, and can adapt to audio and video synchronization tasks with slightly higher subtitle density, and can achieve deep fusion of video audio and video.

[0115] In the above-mentioned audio-visual hard synchronization strategy, when the segment length of the i-th initial audio segment is greater than the subtitle display length of the i-th translated subtitle text, priority is given to acceleration processing, then axis shift processing, and then merging processing; in a possible implementation, when the subtitle density (i.e., the density of the subtitle text) is small or the audio-visual synchronization requirements are not high (i.e., a small amount of audio-visual offset is allowed), priority can also be given to merging processing, then acceleration processing and axis shift processing. Therefore, in a possible implementation, the above-mentioned step S13, according to the segment length of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identification, processes the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text, including:

[0116] Step S1320, when the segment duration of the i-th initial audio segment is greater than the subtitle display duration of the i-th translated subtitle text, based on the subtitle timeline, determining whether the i+1-th translated subtitle text and the i-th translated subtitle text correspond to the same subtitle character identifier;

[0117] Step S1321, when the i-th translated subtitle text and the i+1-th translated subtitle text correspond to the same subtitle character identifier, merging the i-th initial audio segment corresponding to the i-th translated subtitle text and the i+1-th initial audio segment corresponding to the i+1-th translated subtitle text to obtain a first merged audio segment;

[0118] Step S1322, based on the segment length of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identification, the first merged audio segment is processed into a target audio segment that matches the first merged display time; wherein the first merged display time includes the start time of the i-th translated subtitle text display and the end time of the i+1-th translated subtitle text display.

[0119] In a possible implementation, in step S1322, according to the segment length of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the (i+1)-th translated subtitle text, and the subtitle character identifier, the first merged audio segment is processed into a target audio segment matching the first merged display time, including:

[0120] Step S13221: when the segment length of the first merged audio segment is less than or equal to the merged display length corresponding to the first merged display time, determine the first merged audio segment as a target audio segment matching the first merged display time;

[0121] Step S13222, when the segment duration of the first merged audio segment is greater than the merged display duration of the first merged display time, and the number of initial audio segments merged in the first merged audio segment is less than a preset merge quantity threshold, based on the subtitle timeline, determining whether the i+2th translated subtitle text and the i+1th translated subtitle text correspond to the same subtitle character identifier;

[0122] Step S13223, when the (i+2)th translated subtitle text and the (i+1)th translated subtitle text correspond to the same subtitle character identifier, determine the ratio between the segment duration of the first merged audio segment and the combined display duration of the first combined display time as the acceleration ratio corresponding to the first merged audio segment;

[0123] Step S13224, when the acceleration ratio corresponding to the first merged audio segment is less than or equal to the preset acceleration ratio threshold, the first merged audio segment is accelerated according to the acceleration ratio corresponding to the first merged audio segment to obtain a target audio segment that matches the first merged display time.

[0124] In actual applications, after obtaining the target audio segment that matches the first merged display time, the method may further include: step S1323, pushing the target audio segment that matches the first merged display time into the audio queue, and based on aligning the target audio segment that matches the first merged display time to the sixth deviation between the segment cutoff moment on the subtitle timeline and the start moment of display of the i+2th translated subtitle text on the subtitle timeline, filling a blank audio segment corresponding to the sixth deviation after the target audio segment that matches the first merged display time in the audio queue.

[0125] For example, Fig.11 As shown in the figure, the segment duration of the second initial audio segment P2 corresponding to the second translated subtitle text "young lady" is greater than the subtitle display duration of the second translated subtitle text (i.e., the duration between t2 and t3). At this time, based on the subtitle timeline, it is determined that the third translated subtitle text "It's time to get married" corresponds to the same subtitle character identifier "speaker1" as the second translated subtitle text, and the second initial audio segment P2 corresponding to the second translated subtitle text is merged with the third initial audio segment P3 corresponding to the third translated subtitle text to obtain a first merged audio segment P23. If the segment duration of the first merged audio segment P23 is less than or equal to the display duration corresponding to the first merged display time (i.e., the duration between t2 and t5), P23 can be determined as the target audio segment M23 matching the first merged display time, and M23 is pushed into the audio queue, and the segment end time t is aligned to the subtitle timeline (i.e., the start time of M23 is aligned with the start time t2 of the second translated subtitle text) according to M23. M23 The sixth deviation (i.e., t6-t M23 ), fill the blank audio segment (K2) after M23 in the audio queue, where the segment length of K2 is t6-t M23 It should be understood that if M23 is aligned to the end time of the segment on the subtitle timeline, M23 If the sixth deviation from the start time t6 of the display of the fourth translated subtitle text on the subtitle timeline is 0, then a blank audio segment may not be filled after M23 in the audio queue.

[0126] For example, Fig.12As shown, if the duration of the first merged audio segment P23 is greater than the display duration corresponding to the first merged display time (i.e., the duration between t2 and t5), it can be determined whether the number of initial audio segments merged in the first merged audio segment is less than or equal to the preset merge quantity threshold, wherein those skilled in the art can set the specific value of the merge quantity threshold according to actual needs, for example, it can be set to 4, that is, at most 4 initial audio segments are merged; if the first merged audio segment P23 merges two initial audio segments less than the merge quantity threshold (e.g., 4), it can be determined that the fourth translated subtitle text “lady, be Whether "speaker 2" corresponds to the same subtitle character identifier as the third translated subtitle text; if the subtitle character identifier "speaker 3" corresponding to the fourth translated subtitle text is different from the subtitle character identifier "speaker 3" corresponding to the third translated subtitle text, then the ratio between the segment length of the first merged audio segment P23 and the display length of the merged display time (i.e., the duration between t2 and t5) is calculated, and determined as the acceleration ratio corresponding to the first merged audio segment P23; then, it is determined whether the acceleration ratio corresponding to the first merged audio segment P23 is less than or equal to a preset acceleration ratio threshold; if it is less than or equal to the preset acceleration ratio threshold, the first merged audio segment P23 can be accelerated according to the acceleration ratio corresponding to the first merged audio segment P23 (i.e., P23 is doubled in speed according to the acceleration ratio) to obtain a target audio segment M23 matching the first merged display time; then M23 can be pushed to the audio queue, and aligned to the segment cutoff time t on the subtitle timeline according to M23. M23 (At this time t M23 =t5) and the start time t6 of the display of the fourth translated subtitle text on the subtitle time axis (i.e., t6-t M23 =t6-t5), fill the blank audio segment (K2) after M23 in the audio queue, and the segment length of K2 is t6-t M23 Length of time.

[0127] In a possible implementation, in step S1322, according to the segment length of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the (i+1)-th translated subtitle text, and the subtitle character identifier, the first merged audio segment is processed into a target audio segment matching the first merged display time, which may also include:

[0128] Step S13225, when the segment duration of the first merged audio segment is greater than the display duration of the first merged display time and the number of initial audio segments merged in the first merged audio segment is equal to the merge quantity threshold, determine the ratio between the segment duration of the first merged audio segment and the display duration of the first merged display time as the acceleration ratio corresponding to the first merged audio segment;

[0129] Step S13226, when the acceleration ratio corresponding to the first merged audio segment is less than or equal to the preset acceleration ratio threshold, the first merged audio segment is accelerated according to the acceleration ratio corresponding to the first merged audio segment to obtain a target audio segment that matches the first merged display time.

[0130] Exemplarily, if the segment duration of the first merged audio segment P23 is greater than the display duration corresponding to the first merged display time (i.e., the duration between t2 and t5), and the number of initial audio segments merged in the first merged audio segment P23 is equal to the merge quantity threshold (e.g., 2), that is, the number of initial audio segments merged in the first merged audio segment reaches the upper limit, then it is no longer necessary to determine whether the fourth translated subtitle text and the third translated subtitle text correspond to the same subtitle character identifier, and the difference between the segment duration of the first merged audio segment P23 and the display duration of the first merged display time (i.e., the duration between t2 and t5) can be directly calculated. The ratio between the first merged audio segment P23 and the second merged audio segment P24 is determined as the acceleration ratio corresponding to the first merged audio segment P23, and then it is determined whether the acceleration ratio corresponding to the first merged audio segment P23 is less than or equal to the preset acceleration ratio threshold. If the acceleration ratio corresponding to P23 is less than or equal to the preset acceleration ratio threshold, the first merged audio segment P23 can be accelerated according to the acceleration ratio corresponding to the first merged audio segment P23 (that is, P23 is processed at a multiple speed according to the acceleration ratio) to obtain a target audio segment M23 that matches the first merged display time; then M23 can be pushed to the audio queue, and M23 can be aligned to the segment cutoff time t on the subtitle timeline according to the acceleration ratio. M23 (At this time t M23 =t5) and the start time t6 of the display of the fourth translated subtitle text on the subtitle time axis (i.e., t6-t M23 =t6-t5), fill the blank audio segment (K2) after M23 in the audio queue, and the segment length of K2 is t6-t M23 Length of time.

[0131] In a possible implementation, in step S1322, according to the segment length of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the (i+1)-th translated subtitle text, and the subtitle character identifier, processing the first merged audio segment into a target audio segment matching the first merged display time may also include:

[0132] Step S13227, when the acceleration ratio corresponding to the first merged audio segment is greater than the acceleration ratio threshold, the first merged audio segment is accelerated according to the acceleration ratio threshold to obtain a target audio segment corresponding to the first merged display time;

[0133] Step S13228, based on the length offset of the target audio segment corresponding to the first merged display time relative to the merged display time of the first merged display time, compress the blank audio segment before the first merged audio segment in the audio queue, and push the target audio segment corresponding to the first merged display time into the audio queue, wherein the target audio segment pushed into the audio queue corresponds to the segment cutoff time on the subtitle timeline and is aligned with the cutoff time in the first merged display time.

[0134] For example, Fig.13 As shown in FIG. 1 , if the acceleration ratio corresponding to the first merged audio segment P23 is greater than the preset acceleration ratio threshold, the first merged audio segment P23 is accelerated according to the acceleration ratio threshold to obtain the corresponding target audio segment M23; since the acceleration ratio corresponding to the first merged audio segment P23 is greater than the acceleration ratio threshold, the segment duration of the target audio segment M23 after acceleration according to the acceleration ratio threshold is still greater than the merged display duration of the first merged display time (i.e., the duration between t2 and t5), at this time, the difference between the segment duration of the target audio segment M23 and the first merged display duration (i.e., the duration between t2 and t5) can be calculated to obtain the duration deviation. For example, if M23 is 10.5 seconds and the duration between t2 and t5 is 10 seconds, the duration offset is 0.5 seconds; then the blank audio segment before the first merged audio segment in the audio queue can be compressed according to the duration offset, that is, the blank audio segment before the first merged audio segment can be shortened by the duration offset. For example, if the duration offset is 0.5 seconds, the blank audio segment (K1) can be shortened by 0.5 seconds, and then the target audio segment M23 can be pushed into the audio queue. At this time, the target audio segment M23 pushed into the audio queue corresponds to the segment cutoff time t on the subtitle timeline. M23 Aligned with the end time t5 in the combined display time (ie, t M23 =t5).

[0135] In actual applications, the blank audio segment before the first merged audio segment may include any blank audio segment before the first merged audio segment in the audio queue, and is not limited to the previous adjacent blank audio segment; in this method, there is no need to set a maximum compression limit on the compression amount of the blank audio segment, as long as the duration offset requirement of the target audio segment relative to the merged display duration can be met, which is equivalent to moving the target audio segment forward relative to the merged display time.

[0136] It may also happen that the number of initial audio segments merged in the first merged audio segment is less than the merged number threshold. Therefore, in a possible implementation, the method may further include:

[0137] Step S1325: When the segment duration of the first merged audio segment is greater than the merged display duration of the first merged display time, and the number of initial audio segments merged in the first merged audio segment is less than a preset merge quantity threshold, and the i+2th translated subtitle text and the i+1th translated subtitle text correspond to the same subtitle character identifier, merge the first merged audio segment with the i+2th initial audio segment corresponding to the i+2th translated subtitle text to obtain a second merged audio segment;

[0138] Step S1326: Process the second merged audio segment into a target audio segment that matches the second merged display time according to the segment length of the second merged audio segment, the second merged display time corresponding to the i-th translated subtitle text to the i+2-th translated subtitle text, and the subtitle character identifier.

[0139] For example, Fig.14 As shown, if the duration of the first merged audio segment P23 is greater than the display duration corresponding to the first merged display time (i.e., the duration between t2 and t5), it can be determined whether the number of initial audio segments merged in the first merged audio segment is less than or equal to the preset merge quantity threshold; if the first merged audio segment P23 merges two initial audio segments less than the preset merge quantity threshold (e.g., 4), it can be determined whether the fourth translated subtitle text and the third translated subtitle text correspond to the same subtitle character identifier; if the fourth translated subtitle text and the third translated subtitle text correspond to the same subtitle character identifier, then the first merged audio segment P23 can continue to be merged with the fourth initial audio segment corresponding to the fourth translated subtitle text. P4 is merged to obtain a second merged audio segment P234; then the second merged audio segment P234 can be used as a new first merged audio segment, and referring to the processing flow of the first merged audio segment in the above step S1322, the second merged audio segment is processed into a target audio segment M234 that matches the second merged display time. Specifically, if the segment length of the second merged audio segment P234 is less than or equal to the display length corresponding to the second merged display time (that is, the length between t2 and t7), P234 can be determined as the target audio segment M234 that matches the second merged display time, and M234 can be pushed into the audio queue, and aligned to the segment cutoff time t on the subtitle timeline according to M234. M234 The sixth deviation (i.e., t8-t M234 ), fill the blank audio segment (K2) after M234 in the audio queue, where the segment length of K2 is t8-t M234duration; if the segment duration of the second merged audio segment P234 is greater than the display duration corresponding to the second merged display time (i.e. the duration between t2 and t7), then determine whether the number of initial audio segments merged in the second merged audio segment is less than or equal to the preset merge quantity threshold; if less than, then determine whether the 5th translated subtitle text and the 4th translated subtitle text correspond to the same subtitle character identifier; if so, then the second merged audio segment and the 5th initial audio segment corresponding to the 5th translated subtitle text can continue to be merged until the preset merge quantity threshold is reached, or they correspond to different subtitle character identifiers; then, an acceleration ratio calculation can be performed on the merged audio segment formed by merging multiple initial audio segments, and the merged audio segment can be processed into the corresponding target audio segment based on the acceleration ratio, and pushed to the audio queue to fill in blank audio segments and other processing.

[0140] In a possible implementation, it may also happen that the segment duration of the i-th initial audio segment is greater than the subtitle display duration of the i-th translated subtitle text, and the i-th translated subtitle text and the (i+1)-th translated subtitle text correspond to different subtitle character identifiers. Therefore, the above step S13, based on the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier, processes the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text, and may include:

[0141] Step S1327, when the segment duration of the i-th initial audio segment is greater than the subtitle display duration of the i-th translated subtitle text and the i+1-th translated subtitle text corresponds to different subtitle character identifiers, the ratio between the segment duration of the i-th initial audio segment and the subtitle display duration corresponding to the i-th translated subtitle text is determined as the acceleration ratio corresponding to the i-th initial audio segment;

[0142] Step S1328: When the acceleration ratio corresponding to the ith initial audio segment is less than or equal to a preset acceleration ratio threshold, the ith initial audio segment is accelerated according to the acceleration ratio corresponding to the ith initial audio segment to obtain a target audio segment that matches the subtitle display time of the ith translated subtitle text;

[0143] Step S1329, push the target audio segment corresponding to the i-th translated subtitle text into the audio queue, and based on the target audio segment corresponding to the i-th translated subtitle text being aligned to the seventh deviation between the segment cutoff time on the subtitle timeline and the start time of display of the i+1-th translated subtitle text on the subtitle timeline, fill in a blank audio segment corresponding to the seventh deviation after the target audio segment corresponding to the i-th translated subtitle text in the audio queue.

[0144] For example, Fig.15 As shown in the figure, if the fifth initial audio segment P5 corresponding to the fifth translated subtitle text "You should get on the sedan" is greater than the subtitle display duration of the fifth translated subtitle text (i.e., the duration between t8 and t9) and the subtitle character identifier "spraker2" corresponding to the fifth translated subtitle text is the same as the sixth translated subtitle text "Luckystars shine brightly and auspicious times arrive" is different from the subtitle character identifier "spraker3" corresponding to the fifth translated subtitle text. At this time, the ratio between the segment duration of the fifth initial audio segment P5 and the subtitle display duration of the fifth translated subtitle text (i.e., the duration between t8 and t9) can be calculated as the acceleration ratio corresponding to the fifth initial audio segment P5; if the acceleration ratio corresponding to P5 is less than or equal to the preset acceleration ratio threshold, P5 is accelerated according to the acceleration ratio corresponding to P5 to obtain a target audio segment M5 that matches the subtitle display time of the fifth translated subtitle text; then, M5 can be pushed to the audio queue, and based on the segment cutoff time t of the target audio segment M5 aligned to the subtitle timeline (i.e., the start time of M5 is aligned with the start time of the fifth translated subtitle text) M5 (At this time t M5 = t9) and the start time t of the sixth subtitle text display on the subtitle timeline 10 The seventh deviation between 10 -t M5 , fill in the blank audio segment (K3) corresponding to the seventh deviation after M5 in the audio queue, and the segment length of K3 is t 10 -t M5 Length of time.

[0145] In a possible implementation, the i-th initial audio segment is processed into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier, including:

[0146] Step S1330: When the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, the i-th initial audio segment is accelerated according to the acceleration ratio threshold to obtain a target audio segment corresponding to the i-th translated subtitle text;

[0147] Step S1331, based on the duration offset of the target audio segment corresponding to the i-th translated subtitle text relative to the subtitle display duration of the i-th translated subtitle text, compress the blank audio segment before the i-th initial audio segment in the audio queue, and push the target audio segment corresponding to the i-th translated subtitle text into the audio queue, wherein the target audio segment pushed into the audio queue is aligned to the segment cutoff time on the subtitle timeline and the cutoff time in the subtitle display time of the i-th translated subtitle text.

[0148] For example, Fig.16 As shown in the figure, if the segment duration of the initial audio segment P6 corresponding to the sixth translated subtitle text "Lucky stars shine brightly and auspicious times arrive" is greater than the subtitle display duration of the sixth translated subtitle text (ie, t 10 to 11 ) and the subtitle character identifier "spraker3" corresponding to the sixth translated subtitle text is different from the subtitle character identifier "spraker4" corresponding to the seventh translated subtitle text, and the acceleration ratio corresponding to the sixth initial audio segment P6 is greater than the acceleration ratio threshold, then the sixth initial audio segment can be accelerated according to the acceleration ratio threshold to obtain the target audio segment M6 corresponding to the sixth translated subtitle text, and then the segment duration of M6 and the subtitle display duration of the sixth translated subtitle text (i.e., t 10 to 11 The difference between the duration of the segment of M6 and the duration of the subtitle display duration of the sixth translated subtitle text is used as the duration offset of the segment of M6 relative to the subtitle display duration of the sixth translated subtitle text. Then, it can be determined whether the duration offset is less than or equal to the blank audio segment before the sixth initial audio segment in the audio queue (such as Fig.15 K3 in, if the duration offset is less than or equal to the duration of the blank audio segment before the 6th initial audio segment, the blank audio segment before the 6th initial audio segment in the audio queue can be compressed according to the duration offset, that is, the blank audio segment before the 6th initial audio segment is shortened by the duration offset. For example, if the duration offset is 2.5 seconds, and the duration offset of 2.5 seconds is less than or equal to the duration of the blank audio segment K3, the blank audio segment K3 can be compressed by 2.5 seconds, and then the target audio segment M6 can be pushed into the audio queue, wherein, if the duration of the blank audio segment K3 is equal to the duration offset of 2.5 seconds, it is equivalent to removing the blank audio segment K3 from the audio queue, and then the target audio segment M6 can be displayed after being pushed to the audio queue. Fig.16 The arrangement order shown, at this time, the target audio segment M6 pushed into the audio queue corresponds to the segment cutoff time t on the subtitle timeline. M6The end time t of the subtitle display time 11 Alignment (ie t M6 =t 11 ).

[0149] In practical applications, the above steps S1330 to S1331 can be performed when the duration offset of the target audio segment corresponding to the i-th translated subtitle text relative to the subtitle display duration of the i-th translated subtitle text is less than or equal to the duration of the blank audio segment adjacent to the i-th initial audio segment in the audio queue; for the case where the duration offset of the target audio segment corresponding to the i-th translated subtitle text relative to the subtitle display duration of the i-th translated subtitle text is greater than the duration of the blank audio segment adjacent to the i-th initial audio segment in the audio queue, for example, the above Fig.16 The duration offset corresponding to the target audio segment M6 is greater than the duration of the blank audio segment K3. In this case, the i-th initial audio segment can be processed into a target audio segment that matches the subtitle display time of the i-th translated subtitle text with reference to the implementation method of the above steps S1311 to S1312, or the above steps S1313 to S1315. Specifically, the i-th initial audio segment can be directly accelerated according to the acceleration ratio corresponding to the i-th initial audio segment to obtain the target audio segment that matches the subtitle display time of the i-th translated subtitle text and push it into the audio queue to fill the blank audio segment; or, the text content of the i-th translated subtitle text is shortened, and the shortened audio segment corresponding to the shortened i-th translated subtitle text is obtained, and then according to the segment duration of the shortened audio segment, the shortened subtitle display time of the i-th translated subtitle text and the subtitle character identification, the shortened audio segment is processed into a target audio segment that matches the shortened subtitle display time of the i-th translated subtitle text and push it into the audio queue to fill the blank audio segment, and the embodiments of the present disclosure are not limited to this.

[0150] In practical applications, the blank audio segment before the initial audio segment may also include any blank audio segment before the initial audio segment in the audio queue, and is not limited to the previous adjacent blank audio segment; in this method, there is no need to set a maximum compression limit for the compression amount of the blank audio segment, as long as the requirement for the duration offset of the target audio segment relative to the subtitle display duration can be met, which is equivalent to moving the target audio segment forward relative to the subtitle display time.

[0151] Optionally, based on the above steps S1320 to S1331, the present embodiment provides Fig.17 A flowchart of a soft synchronization strategy for audio and video is shown in FIG. Fig.17 As shown, the audio and video soft synchronization strategy includes:

[0152] Step S1701, for the i-th translated subtitle text in the subtitle file, obtaining the i-th initial audio segment Pi corresponding to the i-th translated subtitle text Zi;

[0153] Step S1702, determining whether the duration of the i-th initial audio segment Pi is less than or equal to the subtitle display duration Ti of the i-th translated subtitle text; if so, proceeding to step S1703; if not, proceeding to step S1704;

[0154] Step S1703, determining the i-th initial audio segment Pi as the target audio segment Mi that matches the subtitle display time of the i-th translated subtitle text, and proceeding to step S1709;

[0155] Step S1704, determining whether the i-th translated subtitle text Zi and the i+1-th translated subtitle text Zi+1 correspond to the same subtitle character identifier; if so, proceeding to step S1710; if not, proceeding to step S1705;

[0156] Step S1705, determining whether the acceleration ratio corresponding to the i-th initial audio segment Pi is less than or equal to a preset acceleration ratio threshold; if so, proceeding to step S1706; if not, proceeding to step S1707;

[0157] Step S1706, accelerating the ith initial audio segment Pi according to the acceleration ratio corresponding to the ith initial audio segment Pi, obtaining a target audio segment Mi that matches the subtitle display time of the ith translated subtitle text, and proceeding to step S1709;

[0158] Step S1707, accelerating the i-th initial audio segment Pi according to the acceleration ratio threshold to obtain a target audio segment Mi that matches the subtitle display time of the i-th translated subtitle text;

[0159] Step S1708, compressing the blank audio segment before the i-th initial audio segment in the audio queue based on the duration offset of the target audio segment Mi relative to the subtitle display duration of the i-th translated subtitle text;

[0160] Step S1709, pushing the target audio segment Mi corresponding to the i-th translated subtitle text into the audio queue, and filling a blank audio segment after the target audio segment Mi;

[0161] Step S1710, merging the i-th initial audio segment Pi corresponding to the i-th translated subtitle text and the i+1-th initial audio segment Pi+1 corresponding to the i+1-th translated subtitle text to obtain a first merged audio segment Pii+1;

[0162] Step S1711, determining whether the segment length of the first merged audio segment Pii+1 is less than the merged display time length Tii+1 of the first merged display time; if so, proceeding to step S1712; if not, proceeding to step S1713;

[0163] Step S1712, determining the first merged audio segment Pii+1 as the target audio segment Mii+1 matching the first merged display time, and proceeding to step S1720;

[0164] Step S1713, determining whether the number Nii+1 of initial audio segments merged into the first merged audio segment Pii+1 is less than a preset merged number threshold; if so, proceeding to step S1716; if not, proceeding to step S1714;

[0165] Step S1714, determining whether the i+1th translated subtitle text Zi+1 and the i+2th translated subtitle text Zi+2 correspond to the same subtitle character identifier; if so, proceeding to step S1715; if not, proceeding to step S1716;

[0166] Step S1715, merging the first merged audio segment Pii+1 with the i+2th initial audio segment Pi+2 corresponding to the i+2th translated subtitle text to obtain a second merged audio segment Pii+1+2; then, the second merged audio segment can be used as a new first merged audio segment and the process can be returned to the above step S1711 to execute the above audio-visual soft synchronization process;

[0167] Step S1716, determining whether the acceleration ratio corresponding to the first merged audio segment Pii+1 is less than or equal to a preset acceleration ratio threshold; if so, proceeding to step S1717; if not, proceeding to step S1718;

[0168] Step S1717, according to the acceleration ratio corresponding to the first merged audio segment Pii+1, accelerate the first merged audio segment Pii+1 to obtain the target audio segment Mii+1 corresponding to the first merged display time, and proceed to step S1720;

[0169] Step S1718, accelerating the first merged audio segment Pii+1 according to the acceleration ratio threshold to obtain a first merged display time-matched target audio segment Mii+1;

[0170] Step S1719: compressing the blank audio segment before the first merged audio segment Pii+1 in the audio queue based on the duration offset of the target audio segment Mii+1 relative to the merged display duration of the first merged display time;

[0171] Step S1720: Push the target audio segment Mii+1 corresponding to the first merged display time into the audio queue, and fill a blank audio segment after the target audio segment Mii+1.

[0172] The above-mentioned audio-visual soft synchronization strategy can basically ensure the audio-visual soft synchronization between the audio and video under the same subtitle character identification speaker, and has a slightly lower requirement for the subtitle density. Under the premise of audio-visual synchronization, a small amount of slight audio-visual offset that is not obvious when the same person speaks is allowed. Among them, the number of initial audio clips that can be merged by the audio-visual soft synchronization strategy can be more than the number of initial audio clips that can be merged in the above-mentioned audio-visual hard synchronization strategy. For example, it can be twice the number of initial audio clips that can be merged in the audio-visual hard synchronization strategy, and this is not limited in the embodiments of the present disclosure.

[0173] In step S14, a target audio file synchronized with the video file is generated according to the target audio segments in the subtitle file that match the subtitle display time of each translated subtitle text.

[0174] As described above, an audio queue can be used to store target audio clips and blank audio clips in order, thereby generating a target audio file synchronized with a video file according to the target audio clips in the subtitle file that match the subtitle display time of each subtitle text, including: exporting the target audio clips and blank audio clips arranged in order in the audio queue as a target audio file. The target audio file can be synthesized with the video file and the subtitle file to form video data with subtitles and dubbing, so as to play the video data, thereby achieving an audio-visual effect of synchronized sound and picture.

[0175] According to the audio and video synchronization method of the embodiment of the present invention, by adopting the subtitle display time and subtitle character identification indicated by the subtitle timeline adopted by the original subtitle text of the video file, the initial audio segment corresponding to the translated subtitle text translated from the original subtitle text can be automatically processed into a target audio segment matching the subtitle display time, so that the target audio file synchronized with the video file can be efficiently and quickly generated, and the audio and video synchronization effect can be achieved without human intervention, greatly reducing labor costs.

[0176] As described above, the two audio and video synchronization strategies provided in the above embodiments of the present disclosure, namely, the above steps S1300 to S1315 or Fig.10 The audio and video hard synchronization strategy shown, and the above steps S1320 to S1331 or Fig.17 The audio and video soft synchronization strategies shown in the figure, in practical applications, different application scenarios can be set for the above two audio and video synchronization strategies. Therefore, the embodiment of the present disclosure also provides an audio and video synchronization method, including:

[0177] Step S21, obtaining a subtitle file corresponding to a video file, wherein the subtitle file includes a plurality of translated subtitle texts and a subtitle timeline; the translated subtitle text includes a translation text of an original subtitle text of the video file into a target language; the subtitle timeline is a timeline used by the original subtitle text of the video file, the subtitle timeline represents a subtitle display time of the subtitle text and a subtitle character identifier, the subtitle character identifier represents a character who speaks the subtitle text, and the subtitle display time includes a start time and an end time of the subtitle text display;

[0178] Step S22, when the subtitle density represented by the subtitle time axis is less than or equal to the specified density threshold, execute the above steps S1300 to S1315 or Fig.10 The audio-video hard synchronization strategy and step S14 shown obtain a target audio file synchronized with the video file;

[0179] Step S23, when the subtitle density represented by the subtitle time axis is greater than the specified density threshold, based on the subtitle file, execute the above steps S1320 to S1331 or Fig.17 The illustrated audio-video soft synchronization strategy and step S14 obtain a target audio file synchronized with the video file.

[0180] The subtitle density represents the density of the subtitle text; the denser the subtitle text, the more subtitle text there is and the shorter the time interval between the subtitle texts; it should be understood that those skilled in the art can use a subtitle density detection algorithm known in the art to detect the subtitle density represented on the subtitle time axis, which is not limited in the embodiments of the present disclosure. The specified density threshold can also be customized, which is not limited in the embodiments of the present disclosure.

[0181] In actual applications, the above-mentioned audio and video hard synchronization strategy and the above-mentioned audio and video soft synchronization strategy can be encapsulated as two interfaces respectively. The user can call the required synchronization strategy through the interface according to the actual audio and video synchronization task requirements. For example, when the audio and video synchronization task requires strong synchronization of audio and video, the interface of the audio and video hard synchronization strategy can be called to implement the audio and video hard synchronization logic. When the audio and video synchronization task allows the overall audio and video to have no obvious perceptible offset, the interface of the audio and video soft synchronization strategy can be called to implement the audio and video hard synchronization logic. The embodiments of the present disclosure do not limit this.

[0182] According to the audio and video synchronization method of the embodiment of the present disclosure, the corresponding audio and video synchronization strategy can be automatically selected for different subtitle densities, and the thresholds required for acceleration, shifting, merging and other processing can be adaptively set according to the selected audio and video synchronization strategy; and whether it is the audio and video synchronization strategy selected according to the task adaptation, or the acceleration, shifting, merging and other parameter thresholds adaptively set according to the audio and video synchronization strategy, appropriate fine-tuning can be performed to meet the audio and video requirements. During the entire audio and video synchronization process, no human intervention is required, and audio and video synchronization can be achieved with one click, which greatly reduces labor costs.

[0183] Fig.18 A block diagram of a device for synchronizing audio and video according to an embodiment of the present disclosure is shown. Fig.18 As shown, the device comprises:

[0184] The first acquisition module 181 is used to acquire a subtitle file corresponding to a video file, wherein the subtitle file includes a plurality of translated subtitle texts and a subtitle timeline; the translated subtitle text includes a translation text of the original subtitle text of the video file into a target language; the subtitle timeline is a timeline used by the original subtitle text of the video file, the subtitle timeline represents a subtitle display time of the subtitle text and a subtitle character identifier, the subtitle character identifier represents a character who speaks the subtitle text, and the subtitle display time includes a start time and an end time of the subtitle text display;

[0185] The second acquisition module 182 is used to acquire, for the i-th translated subtitle text in the subtitle file, the i-th initial audio segment corresponding to the i-th translated subtitle text, and determine the subtitle display time and subtitle character identification corresponding to the i-th translated subtitle text based on the subtitle timeline; wherein the i-th initial audio segment is an audio segment generated based on the i-th translated subtitle text;

[0186] A processing module 183 is used to process the i-th initial audio segment into a target audio segment matching the subtitle display time corresponding to the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier;

[0187] The generating module 184 is used to generate a target audio file synchronized with the video file according to the target audio segments in the subtitle file that match the subtitle display time of each translated subtitle text.

[0188] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identification includes: when the segment duration of the i-th initial audio segment is less than or equal to the subtitle display time of the i-th translated subtitle text, determining the i-th initial audio segment as a target audio segment that matches the subtitle display time of the i-th translated subtitle text; wherein the subtitle display time of the i-th translated subtitle text is determined based on the subtitle display time of the i-th translated subtitle text.

[0189] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: when the segment duration of the i-th initial audio segment is greater than the subtitle display time of the i-th translated subtitle text, determining the ratio between the segment duration of the i-th initial audio segment and the subtitle display time of the i-th translated subtitle text as the acceleration ratio corresponding to the i-th initial audio segment; and when the acceleration ratio corresponding to the i-th initial audio segment is less than or equal to a preset acceleration ratio threshold, accelerating the i-th initial audio segment according to the acceleration ratio corresponding to the i-th initial audio segment to obtain the target audio segment that matches the subtitle display time of the i-th translated subtitle text.

[0190] In a possible implementation, after obtaining the target audio segment that matches the subtitle display time of the i-th translated subtitle text, the device also includes: a first push filling module, used to: push the target audio segment into the audio queue, and based on the first deviation between the segment cutoff time when the target audio segment is aligned to the subtitle timeline and the start time when the i+1-th translated subtitle text is displayed on the subtitle timeline, fill the target audio segment in the audio queue with a blank audio segment corresponding to the first deviation; wherein the audio queue is used to store the target audio segment and the blank audio segment in sequence.

[0191] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: when the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, judging whether there are sufficient blank segments before and after the i-th initial audio segment based on the subtitle display time of the i-th translated subtitle text; wherein the existence of sufficient blank segments includes that there are blank segments before and after the i-th initial audio segment and the duration of the blank segments is greater than or equal to the acceleration offset corresponding to the i-th initial audio segment; wherein the acceleration offset includes the difference between the audio segment duration obtained by accelerating the i-th initial audio segment using the acceleration ratio threshold and the duration of the i-th initial audio segment. The difference between the subtitle display durations corresponding to i translated subtitle texts; the duration of the blank segment before the i-th initial audio segment includes the duration of the blank audio segment filled before the i-th initial audio segment; the duration of the blank segment after the i-th initial audio segment includes the duration from the end time of the display of the i-th translated subtitle text to the start time of the display of the i+1-th translated subtitle text on the subtitle timeline; in the case where there are sufficient blank segments before and after the i-th initial audio segment, the i-th initial audio segment is accelerated according to the acceleration ratio threshold to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text; the target audio segment is pushed to the audio queue by occupying the blank segment before the i-th initial audio segment and / or occupying the blank segment after the i-th initial audio segment.

[0192] In a possible implementation, the pushing of the target audio segment into the audio queue by occupying the blank segment before the i-th initial audio segment and / or occupying the blank segment after the i-th initial audio segment comprises: in a case where there are sufficient blank segments before and after the i-th initial audio segment, determining the required compression amount of the blank segment before the i-th initial audio segment and the occupation amount of the blank segment after the i-th initial audio segment by the target audio segment based on a preset offset threshold, the acceleration offset, and the duration of the blank segment before the i-th initial audio segment; wherein the offset threshold is used to limit the maximum compression amount of the blank segment; in a case where the occupation amount is less than or equal to the duration of the blank segment after the i-th initial audio segment, based on the compression amount, compressing the target audio segment after the i-1-th translation word in the audio queue. The method comprises the following steps: compressing the blank audio segment filled after the target audio segment corresponding to the subtitle text; accelerating the i-th initial audio segment according to the acceleration ratio threshold to obtain the target audio segment matching the subtitle display time of the i-th translated subtitle text, and pushing the target audio segment matching the subtitle display time of the i-th translated subtitle text into the audio queue and arranging it after the compressed blank audio segment; based on the target audio segment corresponding to the i-th translated subtitle text in the audio queue being aligned to the second deviation between the segment cutoff time on the subtitle timeline and the start time of the display of the i+1-th translated subtitle text on the subtitle timeline, filling the blank audio segment corresponding to the second deviation after the target audio segment corresponding to the i-th translated subtitle text in the audio queue; wherein the audio queue is used to store the target audio segment and the blank audio segment in sequence.

[0193] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: when the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold and there are no sufficient blank segments before and after the i-th initial audio segment, based on the subtitle timeline, determining whether the i-th translated subtitle text and the (i+1)th translated subtitle text correspond to the same subtitle character identifier; wherein the absence of sufficient blank segments includes the absence of blank segments before and after the i-th initial audio segment, or the duration of the existing blank segments is less than the acceleration offset corresponding to the i-th initial audio segment, or the target audio segment corresponds to the i+1th translated subtitle text. The amount of blank segments after the i-th initial audio segments is greater than the duration of the blank segments after the i-th initial audio segment; when the i-th translated subtitle text and the i+1-th translated subtitle text correspond to the same subtitle character identifier, the i-th initial audio segment corresponding to the i-th translated subtitle text and the i+1-th initial audio segment corresponding to the i+1-th translated subtitle text are merged to obtain a merged audio segment; according to the segment duration of the merged audio segment, the merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier, the merged audio segment is processed into a target audio segment that matches the merged display time; wherein the merged display time includes the start time of the display of the i-th translated subtitle text to the end time of the display of the i+1-th translated subtitle text.

[0194] In a possible implementation, after obtaining the target audio segment that matches the merged display time, the device also includes: a second push filling module, which is used to: push the target audio segment that matches the merged display time into the audio queue, and based on the target audio segment that matches the merged display time being aligned to the third deviation between the segment cutoff time on the subtitle timeline and the start time of display of the i+2th translated subtitle text on the subtitle timeline, fill the audio queue with a blank audio segment corresponding to the third deviation after the target audio segment that matches the merged display time; wherein the audio queue is used to store the target audio segment and the blank audio segment in sequence.

[0195] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: when the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after the i-th initial audio segment, and the i-th translated subtitle text and the i+1-th translated subtitle text correspond to different subtitle character identifiers, processing the i-th initial audio segment according to the acceleration ratio corresponding to the i-th initial audio segment The target audio segment corresponding to the i-th translated subtitle text is accelerated to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text; the target audio segment corresponding to the i-th translated subtitle text is pushed into the audio queue, and based on the target audio segment corresponding to the i-th translated subtitle text being aligned to the fourth deviation between the segment cutoff time on the subtitle timeline and the start time of the display of the i+1-th translated subtitle text on the subtitle timeline, a blank audio segment corresponding to the fourth deviation is filled after the target audio segment corresponding to the i-th translated subtitle text in the audio queue; wherein the audio queue is used to store the target audio segment and the blank audio segment in sequence.

[0196] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: shortening the text content of the i-th translated subtitle text and obtaining a shortened audio segment corresponding to the i-th translated subtitle text when the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after the i-th initial audio segment, and the i-th translated subtitle text and the i+1-th translated subtitle text correspond to different subtitle character identifiers; obtaining a shortened audio segment corresponding to the shortened i-th translated subtitle text according to the segment duration of the shortened audio segment, the shortened The method comprises the following steps: according to the present invention, the target audio segment corresponding to the shortened i-th translated subtitle text is processed into a target audio segment that matches the shortened subtitle display time of the i-th translated subtitle text; the target audio segment corresponding to the shortened i-th translated subtitle text is pushed into an audio queue, and based on the target audio segment corresponding to the shortened i-th translated subtitle text being aligned to the fifth deviation between the segment cutoff time on the subtitle timeline and the start time of the display of the i+1-th translated subtitle text on the subtitle timeline, a blank audio segment corresponding to the fifth deviation is filled after the target audio segment corresponding to the shortened i-th translated subtitle text in the audio queue; wherein the audio queue is used to store the target audio segment and the blank audio segment in sequence.

[0197] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: when the segment duration of the i-th initial audio segment is greater than the subtitle display time of the i-th translated subtitle text, based on the subtitle timeline, determining whether the i+1-th translated subtitle text and the i-th translated subtitle text correspond to the same subtitle character identifier; when the i-th translated subtitle text and the i+1-th translated subtitle text correspond to the same In the case of a subtitle character identification, the i-th initial audio segment corresponding to the i-th translated subtitle text and the i+1-th initial audio segment corresponding to the i+1-th translated subtitle text are merged to obtain a first merged audio segment; according to the segment length of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identification, the first merged audio segment is processed into a target audio segment that matches the first merged display time; wherein the first merged display time includes the start time of the display of the i-th translated subtitle text and the end time of the display of the i+1-th translated subtitle text.

[0198] In a possible implementation, the first merged audio segment is processed into a target audio segment that matches the first merged display time based on the segment duration of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identification, including: when the segment duration of the first merged audio segment is less than or equal to the merged display time corresponding to the first merged display time, the first merged audio segment is determined as the target audio segment that matches the first merged display time.

[0199] In a possible implementation, the processing of the first merged audio segment into a target audio segment matching the first merged display time according to the segment duration of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier includes: when the segment duration of the first merged audio segment is greater than the merged display time of the first merged display time, and the number of merged initial audio segments in the first merged audio segment is less than a preset merged number threshold, judging whether the i+2-th translated subtitle text and the i+1-th translated subtitle text correspond to the same subtitle character identifier based on the subtitle time axis; when the i+2-th translated subtitle text and the i+1-th translated subtitle text correspond to different subtitle character identifiers, determining the ratio between the segment duration of the first merged audio segment and the merged display time of the first merged display time as the acceleration ratio corresponding to the first merged audio segment; and when the acceleration ratio corresponding to the first merged audio segment is less than or equal to a preset acceleration ratio threshold, accelerating the first merged audio segment according to the acceleration ratio corresponding to the first merged audio segment to obtain the target audio segment matching the first merged display time.

[0200] In a possible implementation, the processing of the first merged audio segment into a target audio segment matching the first merged display time according to the segment duration of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier includes: when the segment duration of the first merged audio segment is greater than the merged display time of the first merged display time, and the number of initial audio segments merged in the first merged audio segment is equal to the merged number threshold, determining the ratio between the segment duration of the first merged audio segment and the merged display time of the first merged display time as the acceleration ratio corresponding to the first merged audio segment; and when the acceleration ratio corresponding to the first merged audio segment is less than or equal to a preset acceleration ratio threshold, accelerating the first merged audio segment according to the acceleration ratio corresponding to the first merged audio segment to obtain the target audio segment matching the first merged display time.

[0201] In a possible implementation, the processing of the first merged audio segment into a target audio segment matching the first merged display time according to the segment duration of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier includes: when the acceleration ratio corresponding to the first merged audio segment is greater than the acceleration ratio threshold, accelerating the first merged audio segment according to the acceleration ratio threshold to obtain the target audio segment corresponding to the first merged display time; based on the duration offset of the segment duration of the target audio segment corresponding to the first merged display time relative to the merged display duration of the first merged display time, compressing the blank audio segment in the audio queue before the first merged audio segment, and pushing the target audio segment corresponding to the first merged display time into the audio queue, wherein the segment cutoff time corresponding to the target audio segment pushed into the audio queue on the subtitle timeline is aligned with the cutoff time in the first merged display time.

[0202] In a possible implementation, after obtaining the target audio segment that matches the first merged display time, the device also includes: a third push filling module, which is used to: push the target audio segment that matches the first merged display time into the audio queue, and based on the target audio segment that matches the first merged display time being aligned to the sixth deviation between the segment cutoff time on the subtitle timeline and the start time of display of the i+2th translated subtitle text on the subtitle timeline, fill the audio queue with a blank audio segment corresponding to the sixth deviation after the target audio segment that matches the first merged display time; wherein, the audio queue is used to store the target audio segment and the blank audio segment in sequence.

[0203] In a possible implementation, the method further includes: a merging module, which is used to merge the first merged audio segment with the i+2th initial audio segment corresponding to the i+2th translated subtitle text to obtain a second merged audio segment when the segment duration of the first merged audio segment is greater than the merged display duration of the first merged display time, the number of initial audio segments merged in the first merged audio segment is less than a preset merge quantity threshold, and the i+2th translated subtitle text and the i+1th translated subtitle text correspond to the same subtitle character identifier; the processing module is also used to process the second merged audio segment into a target audio segment matching the second merged display time according to the segment duration of the second merged audio segment, the second merged display time corresponding to the i-th translated subtitle text to the i+2th translated subtitle text, and the subtitle character identifier.

[0204] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: when the segment duration of the i-th initial audio segment is greater than the subtitle display time of the i-th translated subtitle text and the i+1-th translated subtitle text corresponds to different subtitle character identifiers, determining the ratio between the segment duration of the i-th initial audio segment and the subtitle display time corresponding to the i-th translated subtitle text as the acceleration ratio corresponding to the i-th initial audio segment; when the acceleration ratio corresponding to the i-th initial audio segment is less than When the acceleration ratio is greater than or equal to a preset acceleration ratio threshold, the i-th initial audio segment is accelerated according to the acceleration ratio corresponding to the i-th initial audio segment to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text; the target audio segment corresponding to the i-th translated subtitle text is pushed into an audio queue, and based on the target audio segment corresponding to the i-th translated subtitle text being aligned to the seventh deviation between the segment cutoff time on the subtitle timeline and the start time of the display of the i+1-th translated subtitle text on the subtitle timeline, a blank audio segment corresponding to the seventh deviation is filled after the target audio segment corresponding to the i-th translated subtitle text in the audio queue; wherein the audio queue is used to store the target audio segment and the blank audio segment in sequence.

[0205] In a possible implementation, the processing of the i-th initial audio segment into a target audio segment that matches the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: when the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, accelerating the i-th initial audio segment according to the acceleration ratio threshold to obtain the target audio segment corresponding to the i-th translated subtitle text; based on the segment duration offset of the target audio segment corresponding to the i-th translated subtitle text relative to the subtitle display time of the i-th translated subtitle text, compressing a blank audio segment in the audio queue that is located before the i-th initial audio segment, and pushing the target audio segment corresponding to the i-th translated subtitle text into the audio queue, wherein the segment cutoff time of the target audio segment pushed into the audio queue is aligned to the subtitle timeline and the cutoff time in the subtitle display time of the i-th translated subtitle text.

[0206] In a possible implementation, a target audio file synchronized with the video file is generated according to target audio segments in the subtitle file that match the subtitle display time of each subtitle text, including: exporting the target audio segments and blank audio segments arranged in sequence in the audio queue as the target audio file.

[0207] According to the device of the embodiment of the present disclosure, by adopting the subtitle display time and subtitle character identification indicated by the subtitle timeline adopted by the original subtitle text of the video file, the initial audio segment corresponding to the translated subtitle text translated from the original subtitle text can be automatically processed into a target audio segment matching the subtitle display time, so that the target audio file synchronized with the video file can be efficiently and quickly generated, and the audio and video synchronization effect can be achieved without human intervention, greatly reducing labor costs.

[0208] Based on the method of the above embodiment of the present disclosure, the embodiment of the present disclosure also provides another audio and video synchronization device, including:

[0209] A file acquisition module is used to acquire a subtitle file corresponding to a video file, wherein the subtitle file includes a plurality of translated subtitle texts and a subtitle timeline; the translated subtitle text includes a translation text of the original subtitle text of the video file into a target language; the subtitle timeline is a timeline used by the original subtitle text of the video file, the subtitle timeline represents a subtitle display time of the subtitle text and a subtitle character identifier, the subtitle character identifier represents a character who speaks the subtitle text, and the subtitle display time includes a start time and an end time of the subtitle text display;

[0210] The first execution module is used to execute the above steps S1300 to S1315 or S1320 based on the subtitle file when the subtitle density represented by the subtitle timeline is less than or equal to the specified density threshold. Fig.10 The audio-video hard synchronization strategy and step S14 shown obtain a target audio file synchronized with the video file;

[0211] The second execution module is used to execute the above steps S1320 to S1331 or S1341 based on the subtitle file when the subtitle density represented by the subtitle timeline is greater than the specified density threshold. Fig.17 The illustrated audio-video soft synchronization strategy and step S14 obtain a target audio file synchronized with the video file.

[0212] According to the device of the embodiment of the present disclosure, it is possible to automatically select the corresponding audio and video synchronization strategy for different subtitle densities, and can adaptively set the thresholds required for acceleration, shifting, merging and other processing according to the selected audio and video synchronization strategy; and whether it is the audio and video synchronization strategy adaptively selected according to the task, or the acceleration, shifting, merging and other parameter thresholds adaptively set according to the audio and video synchronization strategy, appropriate fine-tuning can be performed to meet the audio and video requirements. During the entire audio and video synchronization process, no human intervention is required, and audio and video synchronization can be achieved with one click, which greatly reduces labor costs.

[0213] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0214] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0215] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0216] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0217] Fig.19 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.19 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0218] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0219] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0220] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0221] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0222] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0223] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0224] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0225] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0226] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0227] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0228] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for synchronizing audio and video, characterized in that: include: Obtaining a subtitle file corresponding to a video file, wherein the subtitle file includes a plurality of translated subtitle texts and a subtitle timeline; The translated subtitle text includes the original subtitle text of the video file translated into the target language; the subtitle timeline is the timeline used by the original subtitle text of the video file, the subtitle timeline represents the subtitle display time of the subtitle text and the subtitle character identification, the subtitle character identification represents the character who speaks the subtitle text, and the subtitle display time includes the start time and the end time of the subtitle text display; For the i-th translated subtitle text in the subtitle file, obtain the i-th initial audio segment corresponding to the i-th translated subtitle text, and determine the subtitle display time and subtitle character identification corresponding to the i-th translated subtitle text based on the subtitle timeline; wherein the i-th initial audio segment is an audio segment generated based on the i-th translated subtitle text; According to the segment length of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier, processing the i-th initial audio segment into a target audio segment matching the subtitle display time corresponding to the i-th translated subtitle text; A target audio file synchronized with the video file is generated according to the target audio segments in the subtitle file that match the subtitle display time of each translated subtitle text.

2. The method according to claim 1, characterized in that The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: In a case where the segment duration of the i-th initial audio segment is less than or equal to the subtitle display duration of the i-th translated subtitle text, the i-th initial audio segment is determined as a target audio segment that matches the subtitle display time of the i-th translated subtitle text; wherein the subtitle display duration of the i-th translated subtitle text is determined based on the subtitle display time of the i-th translated subtitle text.

3. The method according to claim 1, characterized in that The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: When the segment duration of the i-th initial audio segment is greater than the subtitle display duration of the i-th translated subtitle text, a ratio between the segment duration of the i-th initial audio segment and the subtitle display duration of the i-th translated subtitle text is determined as the acceleration ratio corresponding to the i-th initial audio segment; When the acceleration ratio corresponding to the i-th initial audio segment is less than or equal to a preset acceleration ratio threshold, the i-th initial audio segment is accelerated according to the acceleration ratio corresponding to the i-th initial audio segment to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text.

4. The method according to claim 2 or 3, characterized in that: After obtaining the target audio segment that matches the subtitle display time of the i-th translated subtitle text, the method further includes: Pushing the target audio segment into an audio queue, and based on a first deviation between a segment cutoff time when the target audio segment is aligned to the subtitle timeline and a start time when the i+1th translated subtitle text is displayed on the subtitle timeline, filling a blank audio segment corresponding to the first deviation after the target audio segment in the audio queue; The audio queue is used to store target audio segments and blank audio segments in sequence.

5. The method according to claim 3, characterized in that: The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: In a case where the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, judging whether there are sufficient blank segments before and after the i-th initial audio segment based on the subtitle display time of the i-th translated subtitle text; wherein the existence of sufficient blank segments includes that there are blank segments before and after the i-th initial audio segment and the duration of the blank segments is greater than or equal to the acceleration offset corresponding to the i-th initial audio segment; The acceleration offset includes the difference between the duration of the audio segment obtained by accelerating the i-th initial audio segment using the acceleration ratio threshold and the subtitle display duration corresponding to the i-th translated subtitle text; the duration of the blank segment before the i-th initial audio segment includes the duration of the blank audio segment filled before the i-th initial audio segment; the duration of the blank segment after the i-th initial audio segment includes the duration from the end time of displaying the i-th translated subtitle text to the start time of displaying the i+1-th translated subtitle text on the subtitle timeline; In the case where there are sufficient blank segments before and after the i-th initial audio segment, the i-th initial audio segment is accelerated according to the acceleration ratio threshold to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text; The target audio segment is pushed into the audio queue by occupying a blank segment before the i-th initial audio segment and / or occupying a blank segment after the i-th initial audio segment.

6. The method according to claim 5, characterized in that Pushing the target audio segment to the audio queue by occupying a blank segment before the i-th initial audio segment and / or occupying a blank segment after the i-th initial audio segment includes: In the case where there are sufficient blank segments before and after the i-th initial audio segment, based on a preset offset threshold, the acceleration offset and the duration of the blank segment before the i-th initial audio segment, determine the required compression amount of the blank segment before the i-th initial audio segment and the occupation amount of the blank segment after the i-th initial audio segment by the target audio segment; wherein the offset threshold is used to limit the maximum compression amount of the blank segment; When the occupied amount is less than or equal to the duration of the blank segment after the i-th initial audio segment, based on the compression amount, compressing the blank audio segment filled after the target audio segment corresponding to the i-1th translated subtitle text in the audio queue; According to the acceleration ratio threshold, the i-th initial audio segment is accelerated to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text, and the target audio segment that matches the subtitle display time of the i-th translated subtitle text is pushed into the audio queue and arranged after the compressed blank audio segment; Based on the second deviation between the segment cutoff time of the target audio segment corresponding to the i-th translated subtitle text in the audio queue being aligned to the segment cutoff time on the subtitle timeline and the start time of displaying the i+1-th translated subtitle text on the subtitle timeline, filling a blank audio segment corresponding to the second deviation after the target audio segment corresponding to the i-th translated subtitle text in the audio queue; The audio queue is used to store target audio segments and blank audio segments in sequence.

7. The method according to claim 5 or 6, characterized in that: The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: In a case where the acceleration ratio corresponding to the ith initial audio segment is greater than the acceleration ratio threshold and there are no sufficient blank segments before and after the ith initial audio segment, based on the subtitle timeline, it is determined whether the ith translated subtitle text and the (i+1)th translated subtitle text correspond to the same subtitle character identifier; wherein, the absence of sufficient blank segments includes that there are no blank segments before and after the ith initial audio segment, or the duration of the existing blank segments is less than the acceleration offset corresponding to the ith initial audio segment, or the amount of the blank segment after the ith initial audio segment occupied by the target audio segment is greater than the duration of the blank segment after the ith initial audio segment; In a case where the i-th translated subtitle text and the i+1-th translated subtitle text correspond to the same subtitle character identifier, merging the i-th initial audio segment corresponding to the i-th translated subtitle text and the i+1-th initial audio segment corresponding to the i+1-th translated subtitle text to obtain a merged audio segment; According to the segment length of the merged audio segment, the merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identification, the merged audio segment is processed into a target audio segment that matches the merged display time; wherein the merged display time includes the start time of the display of the i-th translated subtitle text to the end time of the display of the i+1-th translated subtitle text.

8. The method according to claim 7, characterized in that After obtaining the target audio segment matching the combined display time, the method further includes: Pushing the target audio segment that matches the combined display time into the audio queue, and based on aligning the target audio segment that matches the combined display time to a third deviation between a segment cutoff moment on the subtitle timeline and a start moment of displaying the i+2 th translated subtitle text on the subtitle timeline, filling a blank audio segment corresponding to the third deviation after the target audio segment that matches the combined display time in the audio queue; The audio queue is used to store target audio segments and blank audio segments in sequence.

9. The method according to claim 7, characterized in that: The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: When the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after the i-th initial audio segment, and the i-th translated subtitle text and the (i+1)-th translated subtitle text correspond to different subtitle character identifiers, the i-th initial audio segment is accelerated according to the acceleration ratio corresponding to the i-th initial audio segment to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text; Pushing the target audio segment corresponding to the i-th translated subtitle text into the audio queue, and based on aligning the target audio segment corresponding to the i-th translated subtitle text to a fourth deviation between a segment cutoff time on the subtitle timeline and a start time of displaying the i+1-th translated subtitle text on the subtitle timeline, filling a blank audio segment corresponding to the fourth deviation after the target audio segment corresponding to the i-th translated subtitle text in the audio queue; The audio queue is used to store target audio segments and blank audio segments in sequence.

10. The method according to claim 7, characterized in that The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: When the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, there are no sufficient blank segments before and after the i-th initial audio segment, and the i-th translated subtitle text and the (i+1)-th translated subtitle text correspond to different subtitle character identifiers, shortening the text content of the i-th translated subtitle text, and obtaining a shortened audio segment corresponding to the shortened i-th translated subtitle text; According to the segment length of the shortened audio segment, the shortened subtitle display time of the i-th translated subtitle text, and the subtitle character identifier, the shortened audio segment is processed into a target audio segment that matches the shortened subtitle display time of the i-th translated subtitle text; Pushing the target audio segment corresponding to the shortened i-th translated subtitle text into the audio queue, and based on aligning the target audio segment corresponding to the shortened i-th translated subtitle text to the fifth deviation between the segment cutoff time on the subtitle timeline and the start time of displaying the i+1-th translated subtitle text on the subtitle timeline, filling a blank audio segment corresponding to the fifth deviation after the target audio segment corresponding to the shortened i-th translated subtitle text in the audio queue; The audio queue is used to store target audio segments and blank audio segments in sequence.

11. The method according to claim 1, characterized in that: The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: In a case where the segment duration of the i-th initial audio segment is greater than the subtitle display duration of the i-th translated subtitle text, judging, based on the subtitle timeline, whether the i+1-th translated subtitle text and the i-th translated subtitle text correspond to the same subtitle character identifier; In a case where the i-th translated subtitle text and the i+1-th translated subtitle text correspond to the same subtitle character identifier, merging the i-th initial audio segment corresponding to the i-th translated subtitle text and the i+1-th initial audio segment corresponding to the i+1-th translated subtitle text to obtain a first merged audio segment; According to the segment length of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identification, the first merged audio segment is processed into a target audio segment that matches the first merged display time; wherein the first merged display time includes the start time of the display of the i-th translated subtitle text and the end time of the display of the i+1-th translated subtitle text.

12. The method according to claim 11, characterized in that The processing of the first merged audio segment into a target audio segment matching the first merged display time according to the segment duration of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier includes: When the segment duration of the first merged audio segment is less than or equal to the merged display duration corresponding to the first merged display time, the first merged audio segment is determined as a target audio segment matching the first merged display time.

13. The method according to claim 11, characterized in that The processing of the first merged audio segment into a target audio segment matching the first merged display time according to the segment duration of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier includes: When the segment duration of the first merged audio segment is greater than the combined display duration of the first combined display time, and the number of initial audio segments merged in the first merged audio segment is less than a preset merge quantity threshold, based on the subtitle timeline, determining whether the (i+2)th translated subtitle text and the (i+1)th translated subtitle text correspond to the same subtitle character identifier; In a case where the (i+2)th translated subtitle text and the (i+1)th translated subtitle text correspond to different subtitle character identifiers, determining a ratio between a segment duration of the first merged audio segment and a combined display duration of the first combined display time as a speedup ratio corresponding to the first merged audio segment; When the acceleration ratio corresponding to the first merged audio segment is less than or equal to a preset acceleration ratio threshold, the first merged audio segment is accelerated according to the acceleration ratio corresponding to the first merged audio segment to obtain a target audio segment that matches the first merged display time.

14. The method according to claim 13, characterized in that The processing of the first merged audio segment into a target audio segment matching the first merged display time according to the segment duration of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier includes: When the segment duration of the first merged audio segment is greater than the combined display duration of the first combined display time, and the number of initial audio segments merged in the first merged audio segment is equal to the combined number threshold, determining a ratio between the segment duration of the first merged audio segment and the combined display duration of the first merged display time as the acceleration ratio corresponding to the first merged audio segment; When the acceleration ratio corresponding to the first merged audio segment is less than or equal to a preset acceleration ratio threshold, the first merged audio segment is accelerated according to the acceleration ratio corresponding to the first merged audio segment to obtain a target audio segment that matches the first merged display time.

15. The method according to claim 13 or 14, characterized in that The processing of the first merged audio segment into a target audio segment matching the first merged display time according to the segment duration of the first merged audio segment, the first merged display time corresponding to the i-th translated subtitle text and the i+1-th translated subtitle text, and the subtitle character identifier includes: When the acceleration ratio corresponding to the first merged audio segment is greater than the acceleration ratio threshold, the first merged audio segment is accelerated according to the acceleration ratio threshold to obtain a target audio segment corresponding to the first merged display time; Based on the duration offset of the target audio segment corresponding to the first merged display time relative to the combined display duration of the first merged display time, the blank audio segment in the audio queue that is located before the first merged audio segment is compressed, and the target audio segment corresponding to the first merged display time is pushed into the audio queue, wherein the target audio segment pushed into the audio queue corresponds to a segment cutoff time on the subtitle timeline that is aligned with the cutoff time in the first merged display time.

16. The method according to any one of claims 11 to 16, characterized in that: After obtaining the target audio segment matching the first combined display time, the method further includes: Pushing the target audio segment matching the first combined display time into the audio queue, and based on aligning the target audio segment matching the first combined display time to the sixth deviation between the segment cutoff moment on the subtitle timeline and the start moment of display of the (i+2)th translated subtitle text on the subtitle timeline, filling a blank audio segment corresponding to the sixth deviation after the target audio segment matching the first combined display time in the audio queue; The audio queue is used to store target audio segments and blank audio segments in sequence.

17. The method according to claim 13, characterized in that The method further comprises: When the segment duration of the first merged audio segment is greater than the combined display duration of the first combined display time, and the number of initial audio segments merged in the first merged audio segment is less than a preset merge quantity threshold, and the (i+2)th translated subtitle text and the (i+1)th translated subtitle text correspond to the same subtitle character identifier, merge the first merged audio segment with the (i+2)th initial audio segment corresponding to the (i+2)th translated subtitle text to obtain a second merged audio segment; According to the segment length of the second merged audio segment, the second merged display time corresponding to the i-th translated subtitle text to the i+2-th translated subtitle text, and the subtitle character identification, the second merged audio segment is processed into a target audio segment that matches the second merged display time.

18. The method according to claim 11, characterized in that The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: In a case where the segment duration of the i-th initial audio segment is greater than the subtitle display duration of the i-th translated subtitle text and the i+1-th translated subtitle text corresponds to different subtitle character identifiers, a ratio between the segment duration of the i-th initial audio segment and the subtitle display duration corresponding to the i-th translated subtitle text is determined as the acceleration ratio corresponding to the i-th initial audio segment; When the acceleration ratio corresponding to the i-th initial audio segment is less than or equal to a preset acceleration ratio threshold, the i-th initial audio segment is accelerated according to the acceleration ratio corresponding to the i-th initial audio segment to obtain a target audio segment that matches the subtitle display time of the i-th translated subtitle text; Pushing the target audio segment corresponding to the i-th translated subtitle text into the audio queue, and based on aligning the target audio segment corresponding to the i-th translated subtitle text to the seventh deviation between the segment cutoff time on the subtitle timeline and the start time of displaying the i+1-th translated subtitle text on the subtitle timeline, filling a blank audio segment corresponding to the seventh deviation after the target audio segment corresponding to the i-th translated subtitle text in the audio queue; The audio queue is used to store target audio segments and blank audio segments in sequence.

19. The method according to claim 18, characterized in that The processing of the i-th initial audio segment into a target audio segment matching the subtitle display time of the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier includes: When the acceleration ratio corresponding to the i-th initial audio segment is greater than the acceleration ratio threshold, the i-th initial audio segment is accelerated according to the acceleration ratio threshold to obtain a target audio segment corresponding to the i-th translated subtitle text; Based on the duration offset of the target audio segment corresponding to the i-th translated subtitle text relative to the subtitle display duration of the i-th translated subtitle text, the blank audio segment in the audio queue that is located before the i-th initial audio segment is compressed, and the target audio segment corresponding to the i-th translated subtitle text is pushed into the audio queue, wherein the target audio segment pushed into the audio queue is aligned to the segment cutoff time on the subtitle timeline and the cutoff time in the subtitle display time of the i-th translated subtitle text.

20. The method according to any one of claims 4, 6, 8-10, 15-19, characterized in that: Generating a target audio file synchronized with the video file according to a target audio segment in the subtitle file that matches the subtitle display time of each subtitle text comprises: Export the target audio segments and blank audio segments arranged in sequence in the audio queue as a target audio file.

21. A method for synchronizing audio and video, characterized in that: include: Obtaining a subtitle file corresponding to a video file, wherein the subtitle file includes a plurality of translated subtitle texts and a subtitle timeline; The translated subtitle text includes the original subtitle text of the video file translated into the target language; the subtitle timeline is the timeline used by the original subtitle text of the video file, the subtitle timeline represents the subtitle display time of the subtitle text and the subtitle character identification, the subtitle character identification represents the character who speaks the subtitle text, and the subtitle display time includes the start time and the end time of the subtitle text display; In the case where the subtitle density represented by the subtitle timeline is less than or equal to the specified density threshold, based on the subtitle file, the method according to any one of claims 2 to 10 and 20 is performed to obtain a target audio file synchronized with the video file; the subtitle density represents the density of the subtitle text; In the case where the subtitle density represented by the subtitle timeline is greater than the specified density threshold, based on the subtitle file, the method of any one of claims 2, 11 to 20 is executed to obtain a target audio file synchronized with the video file.

22. An audio and video synchronization device, characterized in that: include: A first acquisition module, used to acquire a subtitle file corresponding to a video file, wherein the subtitle file includes a plurality of translated subtitle texts and a subtitle timeline; The translated subtitle text includes the original subtitle text of the video file translated into the target language; the subtitle timeline is the timeline used by the original subtitle text of the video file, the subtitle timeline represents the subtitle display time of the subtitle text and the subtitle character identification, the subtitle character identification represents the character who speaks the subtitle text, and the subtitle display time includes the start time and the end time of the subtitle text display; A second acquisition module is used to acquire, for an i-th translated subtitle text in the subtitle file, an i-th initial audio segment corresponding to the i-th translated subtitle text, and determine, based on the subtitle timeline, a subtitle display time and a subtitle character identifier corresponding to the i-th translated subtitle text; wherein the i-th initial audio segment is an audio segment generated based on the i-th translated subtitle text; a processing module, configured to process the i-th initial audio segment into a target audio segment matching the subtitle display time corresponding to the i-th translated subtitle text according to the segment duration of the i-th initial audio segment, the subtitle display time corresponding to the i-th translated subtitle text, and the subtitle character identifier; The generating module is used to generate a target audio file synchronized with the video file according to the target audio segments in the subtitle file that match the subtitle display time of each translated subtitle text.

23. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 22 when executing the instructions stored in the memory.

24. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 22 is implemented.

Images