Method and device for adjusting voice and subtitles, electronic device, and storage medium

By acquiring the playback times of the target audio and subtitles in the video, calculating the time difference, and adjusting the subtitle timeline, the problem of audio and subtitles being out of sync in the video was solved, achieving efficient and accurate synchronization adjustment.

CN116471435BActive Publication Date: 2024-11-29CCTV INT NETWORK CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202310390144.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-12
Publication Date
2024-11-29
Estimated Expiration
2043-04-12

AI Technical Summary

Technical Problem

In existing technologies, when the audio and subtitles in a video are out of sync, users need to manually adjust them, resulting in low adjustment efficiency and difficulty in making accurate adjustments.

Method used

By acquiring the playback times of the target audio and subtitles in the video, calculating the time difference, and adjusting the time axis of the subtitles according to the time difference, the audio and subtitles are synchronized.

Benefits of technology

It improves the efficiency and accuracy of synchronized adjustment of audio and subtitles in videos, avoiding the tedious process of manual adjustment by users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116471435B_ABST
    Figure CN116471435B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for adjusting speech and subtitles, an electronic device, and a storage medium. The method comprises: obtaining a target speech in a video, and determining the first moment of the target speech in the video, wherein the first moment is the moment when the target speech starts to play in the video; extracting a target subtitle matching the target speech from the subtitles of the video according to the first moment and a preset time range; judging whether the target speech and the target subtitle are synchronized according to the first moment and the second moment of the target subtitle in the video, wherein the second moment is the moment when the target subtitle starts to play in the video; in the case where the target speech and the target subtitle are not synchronized, adjusting the time axis of the subtitle in the video according to the time difference between the first moment and the second moment, so as to synchronize the speech and subtitle in the video. The present invention solves the technical problem of using manual adjustment to correct the asynchronism between the sound and subtitle in the video and the poor adjustment effect in the related art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a method and device for adjusting voice and subtitles, an electronic device, and a storage medium. Background Art

[0002] At present, when users watch videos on computers, mobile phones and other electronic devices, they often encounter the situation that the sound and subtitles in the video are not synchronized, that is, the subtitles are ahead of or behind the matching sound for a certain period of time, which seriously affects the user experience. In addition, due to the variety of player types, the asynchrony between sound and subtitles in the video may be caused by various reasons. Therefore, it is difficult to troubleshoot and calibrate the asynchrony between sound and subtitles in the video.

[0003] In the related art, when the subtitles in a video are ahead of or behind, the user needs to manually adjust them based on their feelings, which is troublesome and leads to the problem of adjusting the unsynchronized sound and subtitles in the video, low adjustment efficiency, and difficulty in making accurate adjustments.

[0004] To address the above-mentioned problems, no concise and efficient solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for adjusting voice and subtitles, an electronic device, and a storage medium, so as to at least solve the technical problem of using manual adjustment to correct the asynchronism between the sound and subtitles in a video and the poor adjustment effect in the related art.

[0006] According to one aspect of an embodiment of the present invention, a method for adjusting speech and subtitles is provided, comprising: acquiring a target speech in a video, and determining a first moment of the target speech in the video, wherein the video includes multiple speech lines, and the first moment is the moment when the target speech starts to play in the video; extracting target subtitles matching the target speech from the subtitles of the video according to the first moment and a preset time range; judging whether the target speech is synchronized with the target subtitles according to the first moment and a second moment of the target subtitles in the video, wherein the second moment is the moment when the target subtitles start to play in the video; and in the case where the target speech is not synchronized with the target subtitles, adjusting the time axis of the subtitles in the video according to the time difference between the first moment and the second moment, so as to synchronize the speech in the video with the subtitles.

[0007] Further, based on the first moment and the preset time range, target subtitles matching the target voice are extracted from the subtitles of the video, including: extracting subtitles corresponding to the target preset time range in the video, wherein the target preset time range is determined by the first moment and the preset time range; determining the number of subtitles of the subtitles; in the case where the subtitle is one subtitle, using the subtitle as the target subtitle; in the case where the subtitle is multiple subtitles, determining the target subtitle based on the start playback time of each subtitle in the multiple subtitles in the video, wherein the target subtitle is one of the multiple subtitles, and the start playback time of the target subtitle is the subtitle with the earliest start playback time among the multiple subtitles.

[0008] Furthermore, judging whether the target speech and the target subtitle are synchronized based on the first moment and the second moment of the target subtitle in the video includes: calculating the time difference between the first moment and the second moment; comparing the time difference with a preset threshold to obtain a comparison result, wherein the preset threshold includes at least a first preset threshold and a second preset threshold, the first preset threshold is a negative number, and the second preset threshold is a positive number; judging whether the target speech and the target subtitle are synchronized based on the comparison result.

[0009] Further, based on the comparison result, it is determined whether the target speech is synchronized with the target subtitles, including: when the comparison result is that the time difference is greater than a first preset threshold and less than a second preset threshold, it is determined that the target speech is synchronized with the target subtitles; when the comparison result is that the time difference is less than or equal to the first preset threshold, or the time difference is greater than or equal to the second preset threshold, it is determined that the target speech is not synchronized with the target subtitles.

[0010] Further, obtaining a target voice in a video includes: obtaining multiple candidate voices in the video, wherein the candidate voices are used as candidate voices for the target voice; determining adjacent voice segments of each candidate voice in the video based on a third moment of each candidate voice in the video, wherein the third moment is the moment when each candidate voice starts to play in the video, and the adjacent voice segments include one or more voices; obtaining the start time of the adjacent voice segments of each candidate voice, and determining the time interval between each candidate voice and its adjacent voice segments based on the start time of the adjacent voice segments of each candidate voice and the third moment of each candidate voice; comparing the time interval between each candidate voice and its adjacent voice segments with a preset time interval threshold, and when the time interval is greater than the preset time interval threshold, calculating the score of each candidate voice based on a preset weight, the time interval between each candidate voice and its adjacent voice segments, and the preset time interval threshold; and taking the candidate voice with the largest score among the multiple candidate voices as the target voice.

[0011] Furthermore, adjusting the time axis of the subtitles in the video according to the time difference between the first moment and the second moment also includes: judging whether the time difference is a positive number or a negative number to obtain a judgment result; and adjusting the time axis of the subtitles in the video according to the judgment result and the time difference.

[0012] Furthermore, based on the judgment result and the time difference, the time axis of the subtitles in the video is adjusted, including: calculating the absolute value of the time difference, and taking the absolute value of the time difference as the target time length; when the judgment result indicates that the time difference is a negative number, in the video, the time axis of the subtitles is adjusted backward by the target time length; when the judgment result indicates that the time difference is a positive number, in the video, the time axis of the subtitles is adjusted forward by the target time length.

[0013] According to another aspect of an embodiment of the present invention, another method for adjusting voice and subtitles is provided, comprising: obtaining a first subtitle in a video, and determining a fourth moment of the first subtitle in the video, wherein the video includes multiple voices, and the fourth moment is the moment when the first subtitle starts to play in the video; extracting a first voice matching the first subtitle from the subtitles of the video according to the fourth moment and a preset time range; judging whether the first subtitle is synchronized with the first voice according to the fourth moment and a fifth moment of the first voice in the video, wherein the fifth moment is the moment when the first voice starts to play in the video;

[0014] In the case that the first subtitles are not synchronized with the first voice, the time axis of the subtitles in the video is adjusted according to the time difference between the fourth moment and the fifth moment, so that the voice in the video is synchronized with the subtitles.

[0015] According to another aspect of an embodiment of the present invention, a device for adjusting speech and subtitles is also provided, comprising: a first processing unit, configured to obtain a target speech in a video and determine a first moment of the target speech in the video, wherein the video comprises a plurality of speech lines and the first moment is the moment when the target speech starts to play in the video; a first extraction unit, configured to extract target subtitles matching the target speech from the subtitles of the video according to the first moment and a preset time range; a first judgment unit, configured to judge whether the target speech is synchronized with the target subtitles according to the first moment and a second moment of the target subtitles in the video, wherein the second moment is the moment when the target subtitles start to play in the video; and a first adjustment unit, configured to adjust the time axis of the subtitles in the video according to the time difference between the first moment and the second moment when the target speech is not synchronized with the target subtitles, so as to synchronize the speech in the video with the subtitles.

[0016] Furthermore, the first extraction unit includes: an extraction subunit, used to extract subtitles corresponding to a target preset time range in the video, wherein the target preset time range is determined by the first moment and the preset time range; a first determination subunit, used to determine the number of subtitles of the subtitles; a processing subunit, used to use the subtitles as the target subtitles when the subtitles are one subtitle; a second determination subunit, used to determine the target subtitles according to the start playback time of each subtitle in the video when the subtitles are multiple subtitles, wherein the target subtitle is one of the multiple subtitles, and the start playback time of the target subtitle is the subtitle with the earliest start playback time among the multiple subtitles.

[0017] Furthermore, the first judgment unit includes: a calculation subunit, used to calculate the time difference between the first moment and the second moment; a comparison subunit, used to compare the time difference with a preset threshold to obtain a comparison result, wherein the preset threshold includes at least a first preset threshold and a second preset threshold, the first preset threshold is a negative number, and the second preset threshold is a positive number; the first judgment subunit is used to judge whether the target speech and the target subtitle are synchronized based on the comparison result.

[0018] Furthermore, the judgment subunit includes: a first determination module, used to determine that the target speech is synchronized with the target subtitle when the comparison result is that the time difference is greater than a first preset threshold and less than a second preset threshold; a second determination module, used to determine that the target speech is not synchronized with the target subtitle when the comparison result is that the time difference is less than or equal to the first preset threshold, or that the time difference is greater than or equal to the second preset threshold.

[0019] Furthermore, the first processing unit includes: an acquisition subunit, which is used to acquire the start time of playing the adjacent speech segments of each to-be-selected speech, and determine the time interval between each to-be-selected speech and its adjacent speech segments based on the start time of playing the adjacent speech segments of each to-be-selected speech and the third time of each to-be-selected speech; a third determination subunit, which is used to determine the adjacent speech segments of each to-be-selected speech in the video based on the third time of each to-be-selected speech in the video, wherein the third time is the time when each to-be-selected speech starts playing in the video, and the adjacent speech segments include one or more sentences of speech; the first processing subunit is used to acquire the start time of playing the adjacent speech segments of each to-be-selected speech and the third time of playing the adjacent speech segments of each to-be-selected speech. The first processing unit is used to determine the start playing time of adjacent speech segments, and based on the start playing time of the adjacent speech segments of each to-be-selected speech and the third time of each to-be-selected speech, determine the time interval between each to-be-selected speech and its adjacent speech segments; the second processing subunit is used to compare the time interval between each to-be-selected speech and its adjacent speech segments with a preset time interval threshold, and when the time interval is greater than the preset time interval threshold, calculate the score value of each to-be-selected speech based on the preset weight, the time interval between each to-be-selected speech and its adjacent speech segments, and the preset time interval threshold; and take the to-be-selected speech with the largest score among the multiple to-be-selected speech as the target speech.

[0020] Furthermore, the first adjustment unit also includes: a second judgment subunit, used to judge whether the time difference is a positive number or a negative number, and obtain a judgment result; an adjustment subunit, used to adjust the time axis of the subtitles in the video according to the judgment result and the time difference.

[0021] Furthermore, the adjustment subunit includes: a calculation module, used to calculate the absolute value of the time difference and determine the target time length based on the absolute value of the time difference; a first adjustment module, used to adjust the time axis of the subtitles in the video backward by the target time length when the judgment result indicates that the time difference is a negative number; and a second adjustment module, used to adjust the time axis of the subtitles in the video forward by the target time length when the judgment result indicates that the time difference is a positive number.

[0022] According to another aspect of an embodiment of the present invention, another device for adjusting speech and subtitles is provided, comprising: a second processing unit, configured to obtain a first subtitle in a video and determine a fourth moment when the first subtitle is in the video, wherein the video comprises a plurality of speech sounds and the fourth moment is the moment when the first subtitle starts to play in the video; a second extraction unit, configured to extract a first speech matching the first subtitle from the subtitles of the video according to the fourth moment and a preset time range; a second adjustment unit, configured to determine whether the first subtitle is synchronized with the first speech according to the fourth moment and a fifth moment when the first speech is in the video, wherein the fifth moment is the moment when the first speech starts to play in the video; in the case where the first subtitle is not synchronized with the first speech, adjusting the time axis of the subtitle in the video according to the time difference between the fourth moment and the fifth moment, so as to synchronize the speech in the video with the subtitle.

[0023] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods for adjusting speech and subtitles by executing the executable instructions.

[0024] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned methods for adjusting speech and subtitles.

[0025] In the present invention, by obtaining the target voice in the video, and determining the first moment of the target voice in the video, wherein the video includes multiple voices, and the first moment is the moment when the target voice starts to play in the video; according to the first moment and the preset time range, the target subtitles matching the target voice are extracted from the subtitles of the video; according to the first moment and the second moment of the target subtitles in the video, it is judged whether the target voice and the target subtitles are synchronized, wherein the second moment is the moment when the target subtitles start to play in the video; in the case where the target voice and the target subtitles are not synchronized, according to the time difference between the first moment and the second moment, the time axis of the subtitles in the video is adjusted to synchronize the voice and subtitles in the video. Thus, the technical problem of the manual adjustment method used in the related art to correct the asynchrony between the sound and subtitles in the video and the poor adjustment effect is solved. In the present invention, according to the start playing moment of the target voice in the video and the playing moment of the target subtitles matching the target voice, the time axis of the asynchronously played voice and subtitles is adjusted, so as to avoid the situation where the user manually corrects the asynchrony between the voice and subtitles in the video, and achieves the technical effect of improving the correction efficiency and correction accuracy of the asynchronous voice and subtitles. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 The following is a flow chart of an optional method for adjusting voice and subtitles according to an embodiment of the present invention. Figure 1 ;

[0028] Figure 2 The following is a flow chart of an optional method for adjusting voice and subtitles according to an embodiment of the present invention. Figure 2 ;

[0029] Figure 3 Another optional method for adjusting voice and subtitles according to an embodiment of the present invention is as follows: Figure 3 ;

[0030] Figure 4 is a schematic diagram of an optional voice segment and subtitle according to an embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of an optional matching subtitle selection rule according to an embodiment of the present invention;

[0032] Figure 6 It is a schematic diagram of an optional device for adjusting speech and subtitles according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] The present invention is further described below in conjunction with various embodiments.

[0036] Embodiment 1

[0037] According to an embodiment of the present invention, a method embodiment of an optional method for adjusting speech and subtitles is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 1 The following is a flow chart of an optional method for adjusting voice and subtitles according to an embodiment of the present invention. Figure 1 ,like Figure 1 As shown, the method comprises the following steps:

[0039] Step S101, obtaining a target voice in a video, and determining the first moment of the target voice in the video, wherein the video includes multiple voices, and the first moment is the moment when the target voice starts to play in the video.

[0040] In this embodiment, the speech may be a sound that can be converted into text representation, and the target speech may be the first sentence of a speech segment in a video. In the video, the time interval between the playback time of the target speech and its adjacent speech segments is greater than a preset time interval threshold, that is, the time interval between the playback time of the speech segment where the target speech is located and the adjacent speech segment of the speech segment is greater than the preset time interval threshold.

[0041] The first moment mentioned above can be the moment when the target voice starts to play in the video, that is, the moment when the target voice starts to appear in the video is the first moment of the target voice in the video. For example, if the target voice starts to play at 23 minutes, 16 seconds and 27 milliseconds in the video, then in this embodiment, the first moment is 23 minutes, 16 seconds and 27 milliseconds.

[0042] Step S102: extracting target subtitles matching the target voice from the subtitles of the video according to the first moment and the preset time range.

[0043] The above-mentioned target subtitles can be subtitles that match the above-mentioned target voice, that is, the target subtitles are the subtitles of the target voice in the video, and the target subtitles can be one of one or more subtitles within a time range before and after the first moment. The time range before and after the first moment can be determined by the first moment and a preset time range, that is, the above-mentioned target subtitles can be subtitles in the video within a time range near the first moment.

[0044] Step S103, judging whether the target voice and the target subtitle are synchronized according to the first moment and the second moment of the target subtitle in the video, wherein the second moment is the moment when the target subtitle starts to play in the video.

[0045] The above-mentioned second moment can be the moment when the target subtitle starts to play in the video, that is, the second moment is the moment when the target subtitle starts to appear in the video. In this embodiment, it is possible to determine whether the target voice and the target subtitle are synchronized based on the time difference between the first moment and the second moment of the playback of the target subtitle in the video. Since the situation where the voice and subtitles in the video are out of sync is holistic, that is, when it is determined that a sentence of voice is out of sync with the corresponding subtitle, the voice and subtitles in the video are out of sync. Therefore, judging whether the target voice and the target subtitle are synchronized can determine whether the voice and subtitles in the video are synchronized.

[0046] Step S104, when the target speech and the target subtitles are not synchronized, adjusting the time axis of the subtitles in the video according to the time difference between the first moment and the second moment, so as to synchronize the speech and the subtitles in the video.

[0047] In this embodiment, since the situation where the voice and subtitles in the video are not synchronized is holistic, that is, a subtitle is offset forward or backward by a fixed time from the corresponding voice in the video, all subtitles in the video are also offset forward or backward by a fixed time from the corresponding voice. Therefore, when the target voice and target subtitles are not synchronized, that is, when the voice and subtitles in the video are not synchronized, the time axis of the subtitles in the video can be adjusted according to the time difference between the first moment of the target voice in the video and the second moment of the subtitles in the video to synchronize the voice and subtitles in the video.

[0048] Through the above steps, the time axis of the asynchronously played voice and subtitles is adjusted according to the start playback time of the target voice in the video and the playback time of the target subtitles matching the target voice, thereby avoiding the problem of users manually correcting the asynchronous voice and subtitles in the video, which is not simple and efficient enough, and achieving the technical effect of improving the correction efficiency and correction accuracy of asynchronous voice and subtitles, thereby solving the problem of using manual adjustment to correct the asynchronous sound and subtitles in the video in the related technology, and the technical problem of poor adjustment effect.

[0049] In order to improve the accuracy of matching the extracted subtitles with the target speech, in this embodiment, according to the first moment and the preset time range, the target subtitles that match the target speech in the subtitles of the video also include the following contents: extracting the subtitles corresponding to the target preset time range in the video, wherein the target preset time range is determined by the first moment and the preset time range; determining the number of subtitles; in the case where the subtitles are one subtitle, taking the subtitles as target subtitles; in the case where the subtitles are multiple subtitles, determining the target subtitles according to the start playback time of each subtitle in the multiple subtitles in the video, wherein the target subtitle is one of the multiple subtitles, and the start playback time of the target subtitle is the subtitle with the earliest start playback time among the multiple subtitles.

[0050] The above-mentioned target preset time range can be a time range before and after the first moment, and the time range before and after the first moment can be determined by the first moment and the preset time range, that is, the above-mentioned target preset time range can be a time range in the video within the preset time range near the first moment.

[0051] In this embodiment, subtitles within a target preset time range can be extracted from the video. After the subtitles are extracted, the number of subtitles can also be calculated. When there is only one subtitle, the subtitle can be used as the target subtitle. When there are multiple subtitles, the subtitle with the earliest start time among the multiple subtitles can be used as the target subtitle based on the start time of each subtitle in the video. That is, the subtitle played first within the target preset time range can be used as the target subtitle.

[0052] It should be noted that, in the present embodiment, it is also possible to determine whether subtitles are extracted within the target preset time range in the video. If subtitles are not extracted within the target preset time range, the preset time range can be expanded, that is, the target preset time range is expanded to obtain the expanded target preset time range. Based on the updated target preset time range, the subtitles within the target preset time range are re-extracted from the video. The expanded time range can be the target time range, thereby achieving the technical effect of improving the accuracy of matching the extracted subtitles with the target speech.

[0053] In order to quickly determine whether the voice and subtitles in the video are synchronized, in this embodiment, based on the first moment and the second moment of the target subtitles in the video, it is determined whether the target voice and the target subtitles are synchronized, and the following contents are also included: calculating the time difference between the first moment and the second moment; comparing the time difference with a preset threshold to obtain a comparison result, wherein the preset threshold includes at least a first preset threshold and a second preset threshold, the first preset threshold is a negative number, and the second preset threshold is a positive number; based on the comparison result, determining whether the target voice and the target subtitles are synchronized.

[0054] In this embodiment, since the target subtitles may be delayed from the appearance time of the target voice in the video, and the target subtitles may be earlier than the appearance time of the target voice in the video, the time difference between the first moment and the second moment may be a positive number or a negative number. In order to ensure the accuracy of the result of judging whether the target voice and the target subtitles are synchronized, in this embodiment, the deviation degree between the target voice and the target subtitles can be judged by the size of the time difference or the size of the absolute value of the time difference, that is, the deviation degree between the voice and the subtitles in the video. The larger the size of the time difference or the size of the absolute value of the time difference, the greater the deviation degree between the voice and the subtitles in the video, and the smaller the size of the time difference or the size of the absolute value of the time difference, the smaller the deviation degree between the voice and the subtitles in the video.

[0055] The above-mentioned preset threshold may include a first preset threshold and a second preset threshold, the first preset threshold is a negative number, and the second preset threshold is a positive number. Since the degree of deviation between the subtitles and the voice in the video is small within a certain range, it does not affect the viewing experience of the user. Therefore, when the degree of deviation between the subtitles and the voice in the video is small within a certain range, the voice and subtitles in the video can be synchronized. In this embodiment, by comparing the time difference with the preset threshold, a comparison result is obtained. According to the comparison result, it can be determined whether the degree of deviation between the target subtitles and the target voice is within a certain range, and then it can be determined whether the target voice and the target subtitles are synchronized, thereby achieving a technical effect of quickly determining whether the voice and subtitles in the video are synchronized.

[0056] In order to avoid inaccurate judgment results of whether the speech and subtitles are synchronized, in this embodiment, based on the comparison result, judging whether the target speech and the target subtitles are synchronized also includes the following contents: when the comparison result is that the time difference is greater than the first preset threshold and less than the second preset threshold, it is determined that the target speech and the target subtitles are synchronized; when the comparison result is that the time difference is less than or equal to the first preset threshold, or the time difference is greater than or equal to the second preset threshold, it is determined that the target speech and the target subtitles are not synchronized.

[0057] For example: the above time difference is t, the first preset threshold is a, and the second preset threshold is b. a is a negative number and b is a positive number. When the comparison result is a < t < b, it can be determined that the target voice is synchronized with the target subtitle, that is, the subtitle and the audio (or voice) in the video are originally synchronized; when t ≤ a or t ≥ b, it can be determined that the target voice is not synchronized with the target subtitle, that is, the subtitle and the audio (or voice) in the video are not synchronized, achieving the technical effect of improving the accuracy of the judgment result of whether the voice and the subtitle are synchronized.

[0058] To avoid the situation where the judgment result of whether the voice and the subtitle are synchronized is inaccurate due to the mismatch between the obtained voice and the matched subtitle, in this embodiment, obtaining the target voice in the video further includes the following: obtaining multiple candidate voices in the video, where the candidate voice is a candidate voice for the target voice; based on the third moment of each candidate voice in the video, determining the adjacent voice segments of each candidate voice in the video, where the third moment is the moment when each candidate voice starts to play in the video, and the adjacent voice segments include one or more sentences of voice; obtaining the start playing moment of the adjacent voice segments of each candidate voice, and based on the start playing moment of the adjacent voice segments of each candidate voice and the third moment of each candidate voice, determining the time interval between each candidate voice and its adjacent voice segments; comparing the time interval between each candidate voice and its adjacent voice segments with a preset time interval threshold, and when the time interval is greater than the preset time interval threshold, calculating the score value of each candidate voice based on a preset weight, the time interval between each candidate voice and its adjacent voice segments, and the preset time interval threshold; taking the candidate voice with the largest score value among the multiple candidate voices as the target voice.

[0059] For example: let the start moment of each candidate voice in the video be T1 (corresponding to the above third moment), the moment (start playing moment) when the closest voice segment before this candidate voice occurs be T2, and the moment (start playing moment) when the closest voice segment after this candidate voice occurs be T3. Among them, the above adjacent voice segments correspond to the closest voice segment before the candidate voice, or correspond to the closest voice segment after the candidate voice, and can also correspond to the closest voice segment before the candidate voice and the closest voice segment after the candidate voice.

[0060] If the time interval between T1 and T2, or the time interval between T1 and T3, is greater than the preset threshold value (corresponding to the above-mentioned preset time interval threshold), the selected speech can be selected as "a certain speech" (corresponding to the above-mentioned target speech). It should be noted that the larger the time interval between T1 and T2 or T3, the better the selected speech is as "a certain speech", and the more accurate the subtitles that match the "certain speech" are extracted later, which is conducive to avoiding the problem of misattribution of subtitles and speech, and taking the first speech in a speech as the target speech, achieving the technical effect of improving the accuracy of speech and subtitle synchronization.

[0061] Specifically, the time intervals between the above-mentioned speech to be selected and its preceding and succeeding adjacent speech segments may be counted as it1 and it2, respectively, and the corresponding time interval thresholds may be counted as IT1 and IT2, respectively.

[0062] A speech segment that contains only one sentence of speech and whose time intervals with its preceding and succeeding adjacent speech segments exceed the respective preset time interval thresholds can be preferentially selected as the selected speech, that is, the time interval between the selected speech and its preceding adjacent speech segment exceeds the preset time interval threshold, and the time interval between the selected speech and its succeeding adjacent speech segment exceeds the preset time interval threshold.

[0063] In this embodiment, a speech segment that does not overlap with the preceding and following speech segments in time may be preferentially selected as the speech to be selected.

[0064] Another method of determining the target voice from all the candidate voices is to calculate the score of each candidate voice by calling Formula 1, and then select the candidate voice with the largest score as the target voice.

[0065] w1*[(it1-IT1)] / (it1-IT1+it2-IT2)+w2*[(it2-IT2)] / (it1-IT1+it2-IT2) formula 1

[0066] Among them, w1 and w2 are weights, and the weight values ​​are greater than or equal to 0.

[0067] The judgment of the speech segment can be determined by calculating whether the time intervals between adjacent single-word sounds in the speech are all less than a preset value. If they are all less than p1, it is a speech segment, otherwise it is not a speech segment. In addition, the separation position of each sentence in the speech segment can be determined by the time interval between adjacent single-word sounds being greater than p2 and less than p1, where p2 is less than p1.

[0068] In order to accurately adjust the timeline of subtitles so that the voice and subtitles in the video are accurately synchronized, the timeline of the subtitles in the video is adjusted according to the time difference between the first moment and the second moment, which also includes: judging whether the time difference is a positive number or a negative number to obtain a judgment result; and adjusting the timeline of the subtitles in the video according to the judgment result and the time difference.

[0069] In this embodiment, since the target subtitles may be delayed from the time when the target voice appears in the video, the time difference between the first moment and the second moment may be a positive number; the target subtitles may also be earlier than the time when the target voice appears in the video, so the time difference between the first moment and the second moment may also be a negative number.

[0070] By judging whether the time difference is a positive number or a negative number, a judgment result is obtained; based on the judgment result, the adjustment direction of the time axis of the subtitles in the video can be determined, and based on the time difference, the adjustment time length of the time axis of the subtitles in the video can be determined. For example, the adjustment direction of the subtitle time axis is determined by the positive or negative value of the time difference t. If t is negative, the subtitle time axis is adjusted backward; if t is positive, the subtitle time axis is adjusted forward, thereby achieving the technical effect of accurately adjusting the time axis of the subtitles to achieve accurate synchronization of the voice and subtitles in the video.

[0071] In order to accurately adjust the time axis of the subtitles and synchronize the voice and subtitles in the video, in this embodiment, the time axis of the subtitles in the video is adjusted according to the judgment result and the time difference, and the following contents are also included: calculating the absolute value of the time difference, and determining the target time length according to the absolute value of the time difference; when the judgment result indicates that the time difference is a negative number, in the video, the time axis of the subtitles is adjusted backward by the target time length; when the judgment result indicates that the time difference is a positive number, in the video, the time axis of the subtitles is adjusted forward by the target time length.

[0072] The target time length may be the absolute value of the time difference, or the absolute value of the time difference may be adjusted by a preset time length, and the adjusted value may be used as the target time length. That is, the target time length may be set to be greater than the absolute value of the time difference by a preset time unit length, or the target time length may be set to be less than the absolute value of the time difference by a preset time unit length, based on the user experience of the video subtitle and voice synchronization effect.

[0073] It should be noted that, when the subtitles are independent files, the timeline of the subtitles includes the timeline of the subtitles of the entire video.

[0074] In this embodiment, when the judgment result indicates that the time difference is a negative number, in the video, the timeline of the subtitles can be adjusted backwards by the target time length; when the judgment result indicates that the time difference is a positive number, in the video, the timeline of the subtitles can be adjusted forwards by the target time length, thereby achieving the technical effect of accurately adjusting the timeline of the subtitles and synchronizing the voice and subtitles in the video.

[0075] The present invention is further described below in conjunction with another optional embodiment.

[0076] Embodiment 2

[0077] According to an embodiment of the present invention, another optional method embodiment of a method for adjusting speech and subtitles is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0078] Figure 2 The following is a flow chart of an optional method for adjusting voice and subtitles according to an embodiment of the present invention. Figure 2 ,like Figure 2 As shown, the method comprises the following steps:

[0079] Step S201, obtaining a first subtitle in a video, and determining a fourth moment of the first subtitle in the video, wherein the video includes multiple voices, and the fourth moment is the moment when the first subtitle starts to play in the video;

[0080] Step S202, extracting a first voice matching a first subtitle from the subtitles of the video according to the fourth moment and a preset time range;

[0081] Step S203, judging whether the first subtitle is synchronized with the first voice according to the fourth moment and the fifth moment of the first voice in the video, wherein the fifth moment is the moment when the first voice starts to play in the video;

[0082] Step S204, when the first subtitle is not synchronized with the first voice, adjusting the time axis of the subtitle in the video according to the time difference between the fourth moment and the fifth moment, so as to synchronize the voice in the video with the subtitle.

[0083] This embodiment adopts a method of first selecting a first subtitle and then determining the corresponding voice segment (corresponding to the above-mentioned first voice) to determine whether the voice and subtitle are synchronized, and to perform the synchronization operation of the voice and subtitle. The method is the same as the idea of ​​first determining the target voice and then determining the corresponding target subtitle in the first embodiment. The specific technical means of synchronizing the voice and subtitle in the video are also the same as those in the first embodiment, so they are not repeated here.

[0084] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0085] Embodiment 4

[0086] This embodiment provides another optional method for adjusting speech and subtitles, which is used to detect and correct the problem of speech and subtitle asynchrony in a video.

[0087] Figure 3 The following is a flow chart of an optional method for adjusting voice and subtitles according to an embodiment of the present invention. Figure 3 ,like Figure 3 As shown, the specific steps are:

[0088] Step S01, selecting a certain voice from the video (corresponding to the target voice in the first embodiment);

[0089] Step S02, obtaining the starting time t1 of the speech in the video;

[0090] Step S03, obtaining one or more subtitles that are closest in time to t1, and selecting one subtitle from them according to the rule as the subtitle that matches the selected speech (corresponding to the target subtitle in the first embodiment), and setting the start time of the matching subtitle to t2;

[0091] Step S04, judging whether the sound (i.e., the voice in the video) and the subtitles are synchronized according to the difference between t1 and t2;

[0092] Step S05, if they are not synchronized, adjust the subtitle time axis according to the value of t1-t2 to synchronize the sound with the subtitle;

[0093] Step S06: If synchronized, keep the status quo and do nothing.

[0094] In this embodiment, speech generally refers to speech in human language, which can be expressed in the form of text. Speech can be speech of a person speaking, or speech of human language generated by a machine or intelligent device, because in most cases, only the speech in the video is accompanied by subtitles. It may be necessary to distinguish between human voice and non-human voice, or whether the speech is human language or not.

[0095] Each step from step S01 to step S06 is described in detail below.

[0096] In step S01, how to select "a certain voice" in a video is described:

[0097] Suppose the start time of the selected voice in the video is T1 (corresponding to the third time mentioned above), the time when the voice segment closest to the selected voice occurs (start playing time) is T2, and the time when the voice segment closest to the selected voice occurs (start playing time) is T3, wherein the above-mentioned adjacent voice segments correspond to the voice segment closest to the selected voice, or, correspond to the voice segment closest to the selected voice.

[0098] If the time interval between T1 and T2, or the time interval between T1 and T3, is greater than a preset threshold value (corresponding to the above-mentioned preset time interval threshold value), the speech to be selected is selected as "a certain speech sentence".

[0099] The larger the time interval between T1 and T2 or T3, the more accurate the subtitles that match the "certain voice sentence" selected from the video in the subsequent steps will be when the voice to be selected is taken as the "certain voice sentence", thus avoiding the problem of misattribution of subtitles and voice.

[0100] Because subtitles are displayed one by one, corresponding to each speech segment, taking the first sentence of a speech segment as the subtitle that matches a certain speech segment (corresponding to the target subtitle mentioned above) is more conducive to the synchronization of subtitles and speech, that is, more conducive to the synchronization of subtitles and sound.

[0101] It should be noted that Figure 4 is a schematic diagram of an optional voice segment and subtitle according to an embodiment of the present invention. In this embodiment, the voice segments can be distinguished according to the time interval between the voices in the video, so that Figure 4 There are two speech segments in the figure because the time interval between them is large.

[0102] In this embodiment, it is only necessary to determine whether the time interval between T1 and T2 is greater than a preset threshold value, that is, whether the selected voice is the first sentence of a certain voice segment. If it is greater than the preset threshold value, the selected voice is selected as "a certain voice segment", and the start time of the selected voice is highly likely to correspond to the start time of a certain subtitle.

[0103] In step S02, the start time t1 of a certain speech refers to the time when the speech begins to appear in the video, such as 23 minutes, 16 seconds and 27 milliseconds of the video. Figure 4 In the example, it can be determined that time T1 is the starting time t1 of a certain speech.

[0104] In step S03, the method for obtaining one or more subtitles that are closest to t1 in time is as follows: obtain all subtitles within a preset time range before and after the time t1, that is, extract the corresponding subtitles within the preset time range before and after the time t1 in the video; if no subtitles are obtained within the preset time range before and after the time t1, then expand the preset time range before and after the time t1 and continue the search, and so on.

[0105] Next, an explanation will be given on how to select one subtitle from the subtitles obtained above as the subtitle that matches a certain piece of speech. The rule is as follows: if there is only one subtitle obtained, then use this subtitle as the subtitle that matches a certain piece of speech (corresponding to the target subtitle above); if there are multiple subtitles, then use the subtitle that ranks first in time as the subtitle that matches a certain piece of speech, that is, the subtitle that appears first. This selection method is beneficial for excluding the interference of adjacent subtitles and making the selected subtitle more accurately match a certain piece of speech. Among them, the matching between the subtitle and the speech can mean that this subtitle is prepared for the speech segment where the speech is located. One subtitle is provided for one speech segment.

[0106] Figure 5 is a schematic diagram of an optional subtitle matching rule according to an embodiment of the present invention. As Figure 5 shown, subtitle 1 is not within the preset time range before and after the time t1, while subtitle 2 and subtitle 3 are within the preset time range before and after the time t1, and subtitle 2 is in front of subtitle 3, so subtitle 2 is selected as the matching subtitle for a certain piece of speech, although subtitle 3 is closer to the selected speech.

[0107] Figure 5 In, the start time of subtitle 2 is the start time t2 of the matching subtitle.

[0108] In step S04, the absolute value of the time difference between the time t1 and the time t2 reflects the degree of deviation between a certain piece of speech and the matching subtitle. The larger this absolute value, the greater the deviation. Let the time difference be t;

[0109] t = t1 - t2

[0110] If t < 0, it means that the subtitle lags behind the audio; if t > 0, it means that the subtitle is ahead of the audio.

[0111] It is preset that when a < t < b, it can be determined that the subtitle and the audio were originally synchronized, and then step S06 is executed, that is, no synchronization processing of the subtitle and the audio is performed; otherwise, step S05 is executed to adjust the subtitle time axis to make the sound and the subtitle synchronized.

[0112] a (corresponding to the first preset threshold mentioned above) is a negative number, b (corresponding to the second preset threshold mentioned above) is a positive number, and the selection of the minimum value of a and the maximum value of b can be determined based on experience, which can at least make the user feel that the sound and subtitles are synchronized.

[0113] In step S05, the adjustment of the subtitle time axis refers to the adjustment of the entire subtitle, for example, all subtitles in the video are adjusted forward by 1 second, or adjusted backward by 300 milliseconds.

[0114] The direction of the subtitle time axis adjustment is determined by the positive or negative value of t. If t is negative, the subtitle time axis is adjusted backward; if t is positive, the subtitle time axis is adjusted forward.

[0115] The size of the subtitle timeline adjustment is determined by the absolute value of t, but is not necessarily equal to the absolute value of t. It can be fixed to be slightly smaller or larger than t based on the user experience of the synchronization effect.

[0116] The present invention is further described below in conjunction with another optional embodiment.

[0117] Embodiment 3

[0118] This embodiment provides an optional device for adjusting speech and subtitles. Each implementation unit in the device corresponds to each implementation step in the first embodiment.

[0119] Figure 6 is a schematic diagram of an optional device for adjusting speech and subtitles according to an embodiment of the present invention, such as Figure 6 As shown, the device for adjusting speech and subtitles includes: a first processing unit 51, a first extraction unit 52, a first judgment unit 53, and a first adjustment unit 54.

[0120] Specifically, the first processing unit 51 is used to obtain the target voice in the video and determine the first moment of the target voice in the video, wherein the video includes multiple voices, and the first moment is the moment when the target voice starts to play in the video;

[0121] A first extraction unit 52 is used to extract a target subtitle matching the target voice from the subtitles of the video according to the first moment and a preset time range;

[0122] A first judgment unit 53 is used to judge whether the target voice and the target subtitle are synchronized according to the first moment and a second moment of the target subtitle in the video, wherein the second moment is the moment when the target subtitle starts to play in the video;

[0123] The first adjustment unit 54 is used to adjust the time axis of the subtitles in the video according to the time difference between the first moment and the second moment when the target voice is not synchronized with the target subtitles, so as to synchronize the voice and subtitles in the video.

[0124] In the voice and subtitle adjustment device provided in the fourth embodiment of the present application, the target voice in the video is obtained by the first processing unit 51, and the first moment of the target voice in the video is determined, wherein the video includes multiple voices, and the first moment is the moment when the target voice starts to play in the video, and then the first extraction unit 52 extracts the target subtitle matching the target voice in the subtitle of the video according to the first moment and the preset time range, and the first judgment unit 53 judges whether the target voice and the target subtitle are synchronized according to the first moment and the second moment of the target subtitle in the video, wherein the second moment is the moment when the target subtitle starts to play in the video, and then the first adjustment unit 54 adjusts the time axis of the subtitle in the video according to the time difference between the first moment and the second moment when the target voice and the target subtitle are not synchronized, so as to synchronize the voice and subtitle in the video. This solves the problem of correcting the asynchronism between the sound and subtitle in the video by manual adjustment in the related art, and the technical problem of poor adjustment effect. In the present invention, the time axis of the asynchronously played voice and subtitles is adjusted according to the start playing time of the target voice in the video and the playing time of the target subtitles matching the target voice, thereby avoiding the situation where the user manually corrects the asynchronous voice and subtitles in the video, and achieving the technical effect of improving the correction efficiency and correction accuracy of the asynchronous voice and subtitles.

[0125] Optionally, in the voice and subtitle adjustment device provided in Example 4 of the present application, the first extraction unit 52 includes: an extraction subunit, used to extract subtitles corresponding to a target preset time range in the video, wherein the target preset time range is determined by the first moment and the preset time range; a first determination subunit, used to determine the number of subtitles; a processing subunit, used to use the subtitles as target subtitles when there is one subtitle; a second determination subunit, used to determine the target subtitles when there are multiple subtitles, based on the start playback time of each subtitle in the video, wherein the target subtitle is one of the multiple subtitles, and the start playback time of the target subtitle is the subtitle with the earliest start playback time among the multiple subtitles.

[0126] Optionally, in the speech and subtitle adjustment device provided in Example 4 of the present application, the first judgment unit 53 includes: a calculation subunit, used to calculate the time difference between the first moment and the second moment; a comparison subunit, used to compare the time difference with a preset threshold to obtain a comparison result, wherein the preset threshold includes at least a first preset threshold and a second preset threshold, the first preset threshold is a negative number, and the second preset threshold is a positive number; the first judgment subunit is used to judge whether the target speech and the target subtitle are synchronized based on the comparison result.

[0127] Optionally, in the speech and subtitle adjustment device provided in Example 4 of the present application, the judgment subunit includes: a first determination module, used to determine that the target speech is synchronized with the target subtitle when the comparison result is that the time difference is greater than the first preset threshold and less than the second preset threshold; a second determination module, used to determine that the target speech is not synchronized with the target subtitle when the comparison result is that the time difference is less than or equal to the first preset threshold, or the time difference is greater than or equal to the second preset threshold.

[0128] Optionally, in the voice and subtitle adjustment device provided in Example 4 of the present application, the first processing unit 51 includes: an acquisition subunit, used to acquire the start playback time of the adjacent voice segments of each voice to be selected, and determine the time interval between each voice to be selected and its adjacent voice segments based on the start playback time of the adjacent voice segments of each voice to be selected and the third time of each voice to be selected; a third determination subunit, used to determine the adjacent voice segments of each voice to be selected in the video based on the third time of each voice to be selected in the video, wherein the third time is the time when each voice to be selected starts to play in the video, and the adjacent voice segments include one or more sentences of voice; the first processing subunit , used to obtain the start playback time of the adjacent speech segment of each to-be-selected speech, and determine the time interval between each to-be-selected speech and its adjacent speech segment based on the start playback time of the adjacent speech segment of each to-be-selected speech and the third time of each to-be-selected speech; the second processing subunit is used to compare the time interval between each to-be-selected speech and its adjacent speech segment with a preset time interval threshold, and when the time interval is greater than the preset time interval threshold, calculate the score value of each to-be-selected speech based on the preset weight, the time interval between each to-be-selected speech and its adjacent speech segment, and the preset time interval threshold; the to-be-selected speech with the largest score among multiple to-be-selected speech is used as the target speech.

[0129] Optionally, in the speech and subtitle adjustment device provided in Example 4 of the present application, the first adjustment unit 54 includes: a second judgment subunit, used to judge whether the time difference is a positive number or a negative number, and obtain a judgment result; an adjustment subunit, used to adjust the time axis of the subtitles in the video based on the judgment result and the time difference.

[0130] Optionally, in the speech and subtitle adjustment device provided in Example 4 of the present application, the adjustment subunit includes: a calculation module, used to calculate the absolute value of the time difference, and determine the target time length based on the absolute value of the time difference; a first adjustment module, used to adjust the time axis of the subtitles backward by the target time length in the video when the judgment result indicates that the time difference is a negative number; and a second adjustment module, used to adjust the time axis of the subtitles forward by the target time length in the video when the judgment result indicates that the time difference is a positive number.

[0131] The above-mentioned speech and subtitle adjustment device may also include a processor and a memory. The above-mentioned first processing unit 51, first extraction unit 52, first judgment unit 53 and first adjustment unit 54 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0132] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to adjust the time axis of the asynchronously played voice and subtitles according to the start time of the target voice in the video and the play time of the target subtitles matching the target voice, thereby avoiding the situation where the user manually corrects the asynchronous voice and subtitles in the video, and achieving the technical effect of improving the correction efficiency and correction accuracy of the asynchronous voice and subtitles.

[0133] The above-mentioned memory may include non-permanent memory in a computer-readable medium, a random access memory (RAM) or a non-volatile memory, such as a read-only memory (ROM) or a flash RAM, and the memory includes at least one memory chip.

[0134] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods for adjusting speech and subtitles by executing the executable instructions.

[0135] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned methods for adjusting speech and subtitles.

[0136] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0137] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0142] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for adjusting speech and subtitles, It is characterized in that include: Acquire a target voice in a video, and determine a first moment of the target voice in the video, wherein the video includes multiple voices, the first moment is the moment when the target voice starts to be played in the video, and the time interval between the playing moments of the target voice and its adjacent voice segments is greater than a preset time interval threshold; Extracting target subtitles matching the target voice from the subtitles of the video according to the first moment and the preset time range; Determining whether the target voice and the target subtitle are synchronized according to the first moment and a second moment of the target subtitle in the video, wherein the second moment is the moment when the target subtitle starts to play in the video; When the target speech is not synchronized with the target subtitles, adjusting the time axis of the subtitles in the video according to the time difference between the first moment and the second moment, so as to synchronize the speech with the subtitles in the video; Wherein, according to the first moment and the preset time range, extracting target subtitles matching the target voice from the subtitles of the video includes: extracting subtitles corresponding to the target preset time range from the video, wherein the target preset time range is determined by the first moment and the preset time range, and the target preset time range is a time range before and after the first moment, and the time range before and after the first moment is determined by the first moment and the preset time range; determining the number of subtitles of the subtitles; in the case where the subtitle is one subtitle, taking the subtitle as the target subtitle; in the case where the subtitle is multiple subtitles, determining the target subtitle according to the start playback time of each subtitle in the multiple subtitles in the video, wherein the target subtitle is one of the multiple subtitles, and the start playback time of the target subtitle is the subtitle with the earliest start playback time among the multiple subtitles; Acquiring a target voice in a video, including: acquiring multiple voices to be selected in the video, wherein the voices to be selected are used as voices to be selected for the target voice; determining adjacent voice segments of each voice to be selected in the video based on a third moment of each voice to be selected in the video, wherein the third moment is the moment when each voice to be selected starts to play in the video, and the adjacent voice segments include one or more voices; acquiring the start time of the adjacent voice segments of each voice to be selected, and determining the time interval between each voice to be selected and its adjacent voice segments based on the start time of the adjacent voice segments of each voice to be selected and the third moment of each voice to be selected; comparing the time interval between each voice to be selected and its adjacent voice segments with a preset time interval threshold, and when the time interval is greater than the preset time interval threshold, calculating the score of each voice to be selected based on a preset weight, the time interval between each voice to be selected and its adjacent voice segments, and the preset time interval threshold; taking the voice to be selected with the largest score among the multiple voices to be selected as the target voice, wherein the score of each voice to be selected is determined by formula 1, which is: w1*[(it1-IT1)] / (it1-IT1+it2-IT2)+w2*[(it2-IT2)] / (it1-IT1+it2-IT2)wherein, w1 and w2 represent preset weights, it1 and it2 respectively represent the time interval between the selected speech and its preceding and succeeding adjacent speech segments, and IT1 and IT2 respectively represent the preset time interval thresholds between the selected speech and its preceding and succeeding adjacent speech segments.

2. The adjustment method according to claim 1, It is characterized in that According to the first moment and a second moment of the target subtitle in the video, determining whether the target voice and the target subtitle are synchronized includes: Calculating a time difference between the first moment and the second moment; Compare the time difference with a preset threshold to obtain a comparison result, wherein the preset threshold includes at least a first preset threshold and a second preset threshold, the first preset threshold is a negative number, and the second preset threshold is a positive number; Based on the comparison result, it is determined whether the target speech and the target subtitle are synchronized.

3. The adjustment method according to claim 2, It is characterized in that Based on the comparison result, determining whether the target speech is synchronized with the target subtitles includes: When the comparison result shows that the time difference is greater than a first preset threshold and less than a second preset threshold, determining that the target speech is synchronized with the target subtitle; When the comparison result is that the time difference is less than or equal to the first preset threshold, or the time difference is greater than or equal to the second preset threshold, it is determined that the target speech is not synchronized with the target subtitle.

4. The adjustment method according to claim 1, It is characterized in that According to the time difference between the first moment and the second moment, adjusting the time axis of the subtitles in the video also includes: Determine whether the time difference is a positive number or a negative number, and obtain a determination result; According to the determination result and the time difference, the time axis of the subtitles in the video is adjusted.

5. The adjustment method according to claim 4, It is characterized in that Adjusting the time axis of the subtitles in the video according to the determination result and the time difference includes: Calculating the absolute value of the time difference, and determining the target time length according to the absolute value of the time difference; When the judgment result indicates that the time difference is a negative number, in the video, the time axis of the subtitle is adjusted backwards by the target time length; When the determination result indicates that the time difference is a positive number, in the video, the time axis of the subtitle is adjusted forward by the target time length.

6. A method for adjusting speech and subtitles, It is characterized in that include: Acquire a first subtitle in a video, and determine a fourth moment of the first subtitle in the video, wherein the video includes multiple voices, the fourth moment is the moment when the first subtitle starts to be played in the video, and the time interval between the playing moments of the first subtitle and its adjacent subtitle segments is greater than a preset time interval threshold; Extracting a first voice matching the first subtitle from the voices in the video according to the fourth moment and a preset time range; Determining whether the first subtitle is synchronized with the first voice according to the fourth moment and the fifth moment of the first voice in the video, wherein the fifth moment is the moment when the first voice starts to play in the video; When the first subtitles are not synchronized with the first voice, adjusting the time axis of the subtitles in the video according to the time difference between the fourth moment and the fifth moment, so as to synchronize the voice and subtitles in the video; Wherein, according to the fourth moment and the preset time range, extracting the first voice matching the first subtitle from the voice of the video includes: extracting the voice corresponding to the target preset time range in the video, wherein the target preset time range is determined by the fourth moment and the preset time range, and the target preset time range is a time range before and after the fourth moment, and the time range before and after the fourth moment is determined by the fourth moment and the preset time range; determining the number of the voices; in the case where the voice is one voice, taking the voice as the first voice; in the case where the voice is multiple subtitles, determining the first voice according to the start playback time of each of the multiple voices in the video, wherein the first voice is one of the multiple voices, and the start playback time of the first voice is the voice with the earliest start playback time among the multiple voices; Obtaining a first subtitle in a video, including: obtaining a plurality of subtitles to be selected in the video, wherein the subtitles to be selected are used as subtitles to be selected for the first subtitle; determining adjacent subtitle segments of each subtitle to be selected in the video based on a sixth moment in the video, wherein the sixth moment is a moment when each subtitle to be selected starts to be played in the video, and the adjacent subtitle segments include one or more subtitles; obtaining a start time of playing adjacent subtitle segments of each subtitle to be selected, and determining a time interval between each subtitle to be selected and its adjacent subtitle segments based on the start time of playing adjacent subtitle segments of each subtitle to be selected and the sixth moment of each subtitle to be selected; comparing the time interval between each subtitle to be selected and its adjacent subtitle segments with a preset time interval threshold, and when the time interval is greater than the preset time interval threshold, calculating a score value of each subtitle to be selected based on a preset weight, the time interval between each subtitle to be selected and its adjacent subtitle segments, and the preset time interval threshold; taking a subtitle to be selected with the largest score among the plurality of subtitles to be selected as the first subtitle, wherein the score value of each subtitle to be selected is determined by formula 2, which is: w1*[(it3-IT4)] / (it3-IT4+it2-IT4)+w2*[(it2-IT4)] / (it3-IT4+it2-IT4)wherein, w1 and w2 represent preset weights, it3 and it4 respectively represent the time interval between the subtitle to be selected and its preceding and succeeding adjacent subtitle segments, and IT3 and IT4 respectively represent the preset time interval thresholds between the subtitle to be selected and its preceding and succeeding adjacent subtitle segments.

7. A device for adjusting voice and subtitles, It is characterized in that include: A first processing unit is used to obtain a target voice in a video and determine a first moment of the target voice in the video, wherein the video includes multiple voices, the first moment is a moment when the target voice starts to be played in the video, and a time interval between the playing moments of the target voice and its adjacent voice segments is greater than a preset time interval threshold; A first extraction unit, configured to extract, from the subtitles of the video, a target subtitle matching the target voice according to the first moment and a preset time range; A first judging unit is configured to judge whether the target voice is synchronized with the target subtitle according to the first moment and a second moment when the target subtitle is in the video, wherein the second moment is a moment when the target subtitle starts to play in the video; A first adjustment unit is used to adjust the time axis of the subtitles in the video according to the time difference between the first moment and the second moment when the target voice is not synchronized with the target subtitles, so as to synchronize the voice and subtitles in the video; The first extraction unit includes: an extraction subunit, which is used to extract subtitles corresponding to a target preset time range in the video, wherein the target preset time range is determined by the first moment and the preset time range, and the target preset time range is a time range before and after the first moment, and the time range before and after the first moment is determined by the first moment and the preset time range; a first determination subunit, which is used to determine the number of subtitles of the subtitles; a processing subunit, which is used to use the subtitles as the target subtitles when the subtitles are one subtitle; a second determination subunit, which is used to determine the target subtitles according to the start playback time of each subtitle in the video when the subtitles are multiple subtitles, wherein the target subtitle is one of the multiple subtitles, and the start playback time of the target subtitle is the subtitle with the earliest start playback time among the multiple subtitles; The first processing unit includes: an acquisition subunit, which is used to acquire the target voice in the video, including: acquiring multiple voices to be selected in the video, wherein the voices to be selected are used as voices to be selected for the target voice; a third determination subunit, which is used to determine the adjacent voice segments of each voice to be selected in the video based on the third moment of each voice to be selected in the video, wherein the third moment is the moment when each voice to be selected starts to play in the video, and the adjacent voice segments include one or more sentences of voice; the first processing subunit is used to acquire the start time of the adjacent voice segments of each voice to be selected, and based on the start time of the adjacent voice segments of each voice to be selected The method comprises the following steps: a first processing unit, a second ... w1*[(it1-IT1)] / (it1-IT1+it2-IT2)+w2*[(it2-IT2)] / (it1-IT1+it2-IT2)wherein, w1 and w2 represent preset weights, it1 and it2 respectively represent the time interval between the selected speech and its preceding and succeeding adjacent speech segments, and IT1 and IT2 respectively represent the preset time interval thresholds between the selected speech and its preceding and succeeding adjacent speech segments.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for adjusting speech and subtitles according to any one of claims 1 to 6.

9. An electronic device, It is characterized in that It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the voice and subtitle adjustment method described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Information processing method and electronic device

    CN104795083A