Subtitle generation method and device, electronic equipment and storage medium

Abstract

The invention provides a subtitle generation method and device, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the technical fields of artificial intelligence, natural language processing, signal processing, software engineering, interactive design and the like. The specific implementation scheme comprises the steps of determining a target video; segmenting the audio data of the target video to obtain a plurality of target audio clips; according to a concurrent processing mechanism, performing voice recognition on the plurality of target audio segments to obtain a plurality of text segments; wherein the plurality of text segments are in one-to-one correspondence with the plurality of target audio segments; and translating the plurality of text segments to obtain a target subtitle text. By adopting the method and the device, human resources can be saved, and the generation period of the target subtitle text can be shortened, so that the requirement of acquiring subtitles in real time when a user watches a video in real time is met.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to the fields of artificial intelligence, natural language processing, signal processing, software engineering and interaction design, and specifically to a subtitle generation method, apparatus, electronic device and storage medium. Background Technology

[0002] In the wave of globalization, the globalization of video content is unstoppable. However, the broad multilingual user base is facing the core challenge of "language barriers," which prevents many users from understanding video content without difficulty, greatly restricting the effectiveness of video content dissemination. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for generating subtitles.

[0004] According to a first aspect of this disclosure, a method for generating subtitles is provided, comprising: Identify the target video; The audio data of the target video is segmented to obtain multiple target audio segments; According to the concurrent processing mechanism, speech recognition is performed on multiple target audio segments to obtain multiple text segments; among them, multiple text segments correspond one-to-one with multiple target audio segments; Multiple text segments are translated to obtain the target subtitle text.

[0005] According to a second aspect of this disclosure, a subtitle generation apparatus is provided, comprising: The video determination unit is used to determine the target video; The audio segmentation unit is used to segment the audio data of the target video to obtain multiple target audio segments; The speech recognition unit is used to perform speech recognition on multiple target audio segments according to a concurrent processing mechanism to obtain multiple text segments; wherein, the multiple text segments correspond one-to-one with the multiple target audio segments. The text translation unit is used to translate multiple text segments to obtain the target subtitle text.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; Memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, which are executed by at least one processor to enable the at least one processor to perform the method provided in the first aspect of this disclosure.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions; wherein the computer instructions are used to cause a computer to perform the method provided in the first aspect of this disclosure.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program; wherein, when executed by a processor, the computer program is capable of implementing the method provided in the first aspect of this disclosure.

[0009] Using this disclosure can save human resources and shorten the generation cycle of target subtitle text, so as to meet the user's need to obtain subtitles instantly when watching videos in real time.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A flowchart illustrating a subtitle generation method provided in this embodiment of the disclosure; Figure 2 An example diagram illustrating a method for acquiring a target audio segment provided in an embodiment of this disclosure; Figure 3 A flowchart illustrating a subtitle generation method provided in this embodiment of the disclosure; Figure 4 This is a schematic diagram illustrating an application scenario of a subtitle generation method provided in an embodiment of this disclosure; Figure 5 A schematic structural block diagram of a subtitle generation device provided in this disclosure embodiment; Figure 6 This is a schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] Currently, the cross-language dissemination of video content mainly relies on subtitle translation. However, traditional subtitle generation methods heavily depend on manual processes, requiring at least several independent steps such as "manual transcription (i.e., speech recognition) → manual translation → manual proofreading." This not only consumes a large amount of human resources but also results in an extremely long subtitle generation cycle, making it impossible to meet users' needs for instant subtitles when watching videos in real time (e.g., online courses, live replays, movies, short videos, etc.).

[0014] To address the above problems, this disclosure provides a subtitle generation method that can be applied to electronic devices. The electronic device can be a server, workbench, mainframe computer, or other similar computing device. The following will be combined with... Figure 1 The flowchart shown illustrates a subtitle generation method provided in this disclosure. It should be noted that, although in Figure 1 The flowchart shown illustrates the logical order; however, in some cases, the steps shown or described in the flowchart may be performed in a different order.

[0015] Step S101: Determine the target video.

[0016] The target video can be an online course, a live replay, a movie, a short video, etc., and this disclosure does not limit this.

[0017] Step S102: The audio data of the target video is segmented to obtain multiple target audio segments.

[0018] The audio data may include speech data in a first language and may be mixed with background noise. Here, the first language is the original language of the target video, which may be one of a variety of first candidate languages, such as Chinese, English, Russian, German, French, Italian, Spanish, Japanese, Korean, etc. The background noise may include background music or non-semantic ambient noise (e.g., ambient noise), which is not limited in this embodiment.

[0019] Based on this, it is understood that in this embodiment of the disclosure, the target audio segment may also include speech data in the first language and may be mixed with background noise. Furthermore, in this embodiment of the disclosure, the target audio segment may have a corresponding segment identifier and a timeline identifier. The segment identifier may be a segment code; the timeline identifier may be represented as "T". i ~T i+1 Here, T i It can be used to characterize the start time point of a target audio segment; T i+1 It can be used to characterize the end time of a target audio segment.

[0020] Step S103: According to the concurrent processing mechanism, speech recognition is performed on multiple target audio segments to obtain multiple text segments.

[0021] In other words, in this embodiment of the disclosure, multiple target audio segments can be subjected to concurrent speech recognition to obtain multiple text segments, that is, multiple text segments in the first language. Each text segment corresponds one-to-one with a target audio segment.

[0022] Step S104: Translate multiple text segments to obtain the target subtitle text.

[0023] The target subtitle text can be subtitle text in a second language different from the first language. Here, the second language can be one of several candidate second languages, which may include Chinese, English, Russian, German, French, Italian, Spanish, Japanese, Korean, etc.

[0024] The subtitle generation method provided in this disclosure can determine a target video, segment the audio data of the target video to obtain multiple target audio segments, perform speech recognition on the multiple target audio segments according to a concurrent processing mechanism to obtain multiple text segments, and finally translate the multiple text segments to obtain the target subtitle text. Since this process does not involve manual processes, it not only saves manpower but also shortens the generation cycle of the target subtitle text, meeting the user's need for instant subtitles while watching videos in real time. Moreover, the concurrent processing mechanism used when segmenting the audio data of the target video further shortens the generation cycle of the target subtitle text.

[0025] In some alternative implementations, step S101, namely, "determining the target video", may include: In response to the video translation request, the target video is identified.

[0026] The video translation request can be initiated by the target user via a terminal device, carrying a video identifier. Here, the terminal device can be a conventional computer (e.g., desktop computer, laptop computer, tablet computer, etc.), smartphone, personal digital processor, or other similar computing device; the video identifier can be the video name, video encoding, etc.

[0027] After receiving a video translation request, the electronic device can extract the video identifier from the request and, based on the identifier, determine the target video from a pool of candidate videos. These candidate videos can be existing video content stored in the target cloud storage space. Here, the target cloud storage space can be the target user's cloud storage space.

[0028] Through the above methods, in this embodiment of the disclosure, a target video can be determined in response to a video translation request, thereby improving the controllability of the autonomous determination of the target video. In other words, this embodiment of the disclosure can receive explicit video translation requests and directly specify the processing object of the subtitle generation method based on the video translation request, that is, directly specify the target video, thereby achieving accurate matching between the target video and user needs and avoiding unnecessary waste of computing resources.

[0029] In some alternative implementations, step S101, namely, "determining the target video", may also include: The first candidate video has been determined; Obtain the original language category of the first candidate video; If the original language category differs from the reference language category, the first candidate video will be used as the target video.

[0030] The first candidate video can be the latest video content in the target cloud storage space.

[0031] In this embodiment of the disclosure, the original language category may be Chinese, English, Russian, German, French, Italian, Spanish, Japanese, Korean, etc.; the reference language category may be the target user's commonly used language category (e.g., the target user's native language category), which may be set by the target user or determined based on historical subtitle generation events triggered by the target user. This embodiment of the disclosure does not impose any restrictions on this.

[0032] Furthermore, it should be noted that in this embodiment of the disclosure, the first candidate video can be used as the target video when the original language category is different from the reference language category and the current time period is nighttime. The nighttime period can be set according to application requirements; for example, it can be set to 00:00 to 8:00 AM. This embodiment of the disclosure does not impose any restrictions on this.

[0033] Through the above methods, in this embodiment of the disclosure, a first candidate video can be determined, and its original language category can be obtained. If the original language category differs from the reference language category, the first candidate video is used as the target video. In other words, in this embodiment of the disclosure, by anticipating user needs, the subtitle generation process can be transformed from passive "synchronous processing" to active "asynchronous computation," allowing users to directly obtain instantaneous results when they have an actual subtitle acquisition request. This effectively solves the latency and resource contention problems caused by synchronous processing in high-concurrency scenarios.

[0034] After the target video is determined, step S102 can be executed, that is, the audio data of the target video is segmented to obtain multiple target audio segments.

[0035] In this embodiment of the disclosure, step S102, as a first optional implementation, may include: The audio data of the target video is segmented according to the preset segment length to obtain multiple target audio segments.

[0036] The preset segment length can be set according to application requirements, for example, it can be set to 30 seconds (S), and this embodiment does not limit it; the target audio segment can be in waveform sound file (WAV) format.

[0037] In this embodiment of the present disclosure, the audio data of a target video can be segmented according to a preset segment length to obtain multiple target audio segments. Since this process does not involve complex calculations, it can improve the efficiency of acquiring target audio segments.

[0038] In this embodiment of the disclosure, step S102, as a second optional implementation, may also include: The audio data is segmented according to the pause intervals to obtain multiple initial audio segments; The voiceprint features of each initial audio segment are obtained to obtain multiple voiceprint features; Based on multiple voiceprint features, at least some of the initial audio segments from multiple initial audio segments are merged to obtain multiple target audio segments.

[0039] The pause interval position can be the position of the audio track in the audio data where the speech pause duration exceeds the preset pause duration. Here, the preset pause duration can be set according to application requirements, for example, it can be set to 2 seconds, and this embodiment of the present disclosure does not limit it.

[0040] Furthermore, in this embodiment of the disclosure, the voiceprint feature is the voice fingerprint, which can be a fixed-length, low-dimensional numerical vector. In one example, a voiceprint recognition model can be used to obtain the voiceprint features of each initial audio segment to obtain multiple voiceprint features. The voiceprint recognition model can be, but is not limited to, an Emphasized Channel Attention, Propagation And Aggregation ForTime Delay Neural Network (ECAPA-TDNN), a Residual Network with Squeeze-and-Excitation Module (ResNetSE), an Enhanced Res2Net with Local and Global Feature Fusion (ERes2Net), or a Gaussian Mixture Model - Universal Background Model (GMM-UBM). This embodiment of the disclosure does not limit the specific implementation of the model.

[0041] After obtaining multiple voiceprint features, at least some of the initial audio segments can be merged based on these features to obtain multiple target audio segments. For example, it is possible to: Based on multiple voiceprint features, adjacent initial audio segments with similar voiceprint features are clustered into an audio segment group to obtain multiple audio segment groups; Each audio segment group is used as a target segment group, and the total duration of the first segment of the initial audio segments included in the target segment group is obtained; If the total duration of the first segment does not exceed the preset duration threshold, the initial audio segments included in the target segment group are merged to obtain the target audio segment. Alternatively, if the total duration of the first segment exceeds a preset duration threshold, the target segment group is split into multiple lower-level audio groups; the initial audio segments included in each lower-level audio group are merged to obtain the target audio segment.

[0042] In one example, for two adjacent initial audio segments, the cosine similarity of their voiceprint features can be calculated as the first similarity. If the first similarity is greater than or equal to a preset similarity threshold, it is determined that the voiceprint features of the two initial audio segments are similar (that is, the two initial audio segments can be considered to correspond to the same speaker), and then the two initial audio segments are clustered into an audio segment group. The preset similarity threshold can be set according to application requirements, for example, it can be set to 0.85, and this embodiment of the present disclosure does not limit it.

[0043] After obtaining multiple audio segment groups, each audio segment group can be used as a target segment group. The total duration of the first segment of the initial audio segments included in the target segment group can be obtained. For example, the lengths of the initial audio segments included in the target segment group can be superimposed to obtain the total duration of the first segment. Subsequently, if the total duration of the first segment does not exceed a preset duration threshold, the initial audio segments included in the target segment group can be merged to obtain the target audio segment; or, if the total duration of the first segment exceeds the preset duration threshold, the target segment group can be split to obtain multiple lower-level audio groups, and the initial audio segments included in each lower-level audio group can be merged to obtain the target audio segment. The total duration of the second segment of the initial audio segments included in the lower-level audio groups does not exceed the preset duration threshold. Here, the preset duration threshold can be set according to application requirements, for example, it can be set to 30 seconds, and this embodiment does not limit this; the total duration of the second segment can be obtained by superimposing the lengths of the initial audio segments included in the lower-level audio groups.

[0044] Please combine Figure 2 For example, the audio data is segmented according to the pause intervals to obtain six initial audio segments: initial audio segment A1, initial audio segment A2, initial audio segment A3, initial audio segment A4, initial audio segment A5, and initial audio segment A6. Then, based on multiple voiceprint features that correspond one-to-one with the multiple initial audio features, adjacent initial audio segments with similar voiceprint features are clustered into audio segment groups, resulting in audio segment group B1 and audio segment group B2. Audio segment group B1 includes initial audio segments A1 and A2; audio segment group B2 includes initial audio segments A3, A4, A5, and A6.

[0045] Assuming that the total duration of the first segment of audio segment group B1 does not exceed the preset duration threshold, the initial audio segments (i.e., initial audio segments A1 and A2) included in audio segment group B1 can be merged to obtain the target audio segment C1. Assuming that the total duration of the first segment of audio segment group B2 exceeds the preset duration threshold, audio segment group B2 can be split to obtain the lower audio group B21 (e.g., including initial audio segments A3 and A4) and the lower audio group B22 (e.g., including initial audio segments A5 and A6). The initial audio segments included in the lower audio group B21 can be merged to obtain the target audio segment C2, and the initial audio segments included in the lower audio group B22 can be merged to obtain the target audio segment C3.

[0046] In this embodiment of the present disclosure, audio data can be segmented according to pause intervals to obtain multiple initial audio segments. The voiceprint features of each initial audio segment are then acquired to obtain multiple voiceprint features. Based on these multiple voiceprint features, at least some of the initial audio segments are merged to obtain multiple target audio segments. This intelligent segmentation method, based on "speaker-organized content," achieves a qualitative leap from "physical segmentation" to "semantic segmentation." That is, instead of mechanically merging the speech content of different speakers into the same target audio segment, it intelligently clusters continuous sentences from the same speaker into the same target audio segment, providing rich and reliable contextual information for subsequent speech recognition, thereby improving the accuracy of speech recognition.

[0047] Furthermore, when merging at least some initial audio segments from multiple initial audio segments based on multiple voiceprint features to obtain multiple target audio segments, adjacent initial audio segments with similar voiceprint features can be clustered into an audio segment group based on multiple voiceprint features to obtain multiple audio segment groups. Each audio segment group is then used as a target segment group. The total duration of the first segment of the initial audio segments included in the target segment group is obtained so that, if the total duration of the first segment does not exceed a preset duration threshold, the initial audio segments included in the target segment group are merged to obtain the target audio segment. Alternatively, if the total duration of the first segment exceeds the preset duration threshold, the target segment group is split to obtain multiple lower-level audio groups, and the initial audio segments included in each lower-level audio group are merged to obtain the target audio segment. In other words, this embodiment of the disclosure achieves an optimal balance between intelligent speech recognition accuracy and timeliness. Specifically, it prioritizes the continuity of content from the same speaker through voiceprint clustering, thereby improving speech recognition accuracy. Simultaneously, it introduces a duration threshold mechanism to prevent excessively long target audio segments from being generated due to prolonged speaking by a single person, thus ensuring appropriate granularity for individual speech recognition tasks. This dynamic adjustment strategy not only improves speech recognition accuracy from the source but also optimizes backend concurrent processing efficiency, ultimately achieving a dual improvement in both speech recognition accuracy and timeliness.

[0048] After identifying the target video and segmenting its audio data to obtain multiple target audio segments, step S103 can be executed, that is, according to the concurrent processing mechanism, speech recognition is performed on the multiple target audio segments to obtain multiple text segments.

[0049] In this embodiment of the disclosure, step S103, as a first optional implementation, may include: Obtain the recognition difficulty level of the target video; Based on the recognition difficulty level, the first target model is determined from multiple candidate recognition models; Following the concurrent processing mechanism, the first target model is used to perform speech recognition on multiple target audio segments to obtain multiple text segments.

[0050] The difficulty level can be high, medium, or low, and can be determined based on the video category of the target video.

[0051] (1) For higher difficulty levels, it can correspond to: Professional and academic categories, such as research papers and reports, high-level academic conferences, etc. Legal and judicial matters, such as court trials and interpretation of legal provisions; Healthcare-related, such as medical seminars and diagnostic discussions; Complex acoustic environments, such as multi-person debates and noisy live interviews; Important business meetings, such as transnational negotiations and strategic decision-making meetings; Categories of dialect communication, such as content featuring local characteristics and interviews with specific groups.

[0052] (2) For the medium difficulty level, it can correspond to: Academic lectures, such as university courses and technology sharing sessions; Business and finance related, such as financial analysis and business interviews; Information Technology (IT) category, such as technology launch events, programming tutorials, etc.; Film genres, such as science fiction films, horror films, etc.; Narration for documentaries, such as nature and history documentaries (clear pronunciation, but professional content); Standard meeting minutes, such as those for company meetings and project discussions.

[0053] (3) For lower difficulty levels, the corresponding levels are: Everyday conversations, such as video blogs, lifestyle sharing, and daily interviews; Clear and concise reporting, such as weather forecasts and standard news briefings; Simple educational materials, such as basic education content and introductory skills tutorials; Entertainment and leisure content, such as funny short videos and game live streams; Telephone recordings, such as customer service recordings and clear call logs.

[0054] In this embodiment, when the target video's recognition difficulty level is high, a large-parameter model is selected from multiple candidate recognition models as the first target model; when the target video's recognition difficulty level is medium, a medium-parameter model is selected from multiple candidate recognition models as the first target model; and when the target video's recognition difficulty level is low, a small-parameter model is selected from multiple candidate recognition models as the first target model. The large-parameter model can be a Whisper speech-to-text model with a total of 1550 megabytes (MB); the medium-parameter model can be a Whisper model with a total of 796 MB; and the small-parameter model can be a Whisper model with a total of 244 MB.

[0055] After determining the first target model, a concurrent processing mechanism can be used to perform speech recognition on multiple target audio segments to obtain multiple text segments. In one example, a first target number of first recognition threads can be started, and the multiple target audio segments can be relatively evenly distributed among the first target number of recognition threads, so that each first recognition thread calls the first target model to perform speech recognition on the obtained target audio segments to obtain the corresponding text segments. The first target number can be set according to application requirements, for example, it can be set to 12, and this embodiment of the disclosure does not limit this; the text segments can be in subtitle (SRT) format.

[0056] For example, if there are 240 target audio segments and 12 first recognition threads, the 240 target audio segments can be relatively evenly distributed among the 12 first recognition threads. That is, each first recognition thread obtains 20 target audio segments. Then, for each first recognition thread, the first recognition thread can perform speech recognition on its 20 target audio segments to obtain 20 text segments that correspond one-to-one with these 20 target audio segments.

[0057] Through the above methods, in this embodiment of the disclosure, the recognition difficulty level of the target video can be obtained. Based on the recognition difficulty level, a first target model is determined from multiple candidate recognition models. Then, according to the concurrent processing mechanism, the first target model is used to perform speech recognition on multiple target audio segments to obtain multiple text segments. In other words, in this embodiment of the disclosure, by finely dividing the recognition difficulty level, stronger computing resources (e.g., large parameter models) can be intelligently allocated to high-difficulty video content to ensure the accuracy of transcription of key information; at the same time, medium-intensity computing resources (e.g., medium parameter models) are allocated to medium-difficulty video content, while lightweight models (e.g., small parameter models) are used for low-difficulty video content. This "on-demand allocation of computing resources" mechanism can achieve an optimal balance between resource scheduling and quality efficiency.

[0058] In this embodiment of the disclosure, step S103, as a second optional implementation, may include: Obtain a recognition difficulty score for each target audio segment; For each target audio segment, based on the recognition difficulty score of the target audio segment, a second target model corresponding to the target audio segment is determined from multiple candidate recognition models; According to the concurrent processing mechanism, multiple second target models, which correspond one-to-one with multiple target audio segments, are used to perform speech recognition on multiple target audio segments to obtain multiple text segments.

[0059] The recognition difficulty score characterizes the speech recognition difficulty of a target audio segment, and the two are positively correlated. In one example, the recognition difficulty score for each target audio segment can be obtained as follows: Define multiple evaluation dimensions; For each target audio segment, the recognition difficulty of the target audio segment is evaluated from multiple evaluation dimensions to obtain multiple scores to be fused; based on the multiple scores to be fused, the recognition difficulty score of the target audio segment is obtained.

[0060] The evaluation dimensions may include at least one of the first, second, and third dimensions.

[0061] In this embodiment of the disclosure, the first dimension corresponds to the signal-to-noise ratio (SNR) score to be fused; the second dimension corresponds to the speech speed score to be fused; and the third dimension corresponds to the speech fundamental frequency variance score to be fused.

[0062] Signal-to-noise ratio (SNR) can be used to characterize the ratio of the signal strength of speech data to the signal strength of background noise in the target audio segment. The lower the SNR, the greater the difficulty of speech recognition. Speech velocity can be the average number of words in the target audio segment per unit time (e.g., 30 seconds). The higher the speech velocity, the greater the difficulty of speech recognition. Speech fundamental frequency variance can be used to characterize the degree of pitch variation in the target audio segment. Both excessively small and excessively large speech fundamental frequency variances will increase the difficulty of speech recognition. Specifically, a small speech fundamental frequency variance indicates a flat and monotonous tone, lacking rhythmic variation. This may make it difficult for the recognition model to distinguish between interrogative and declarative sentences, or even word boundaries, thus increasing the difficulty of speech recognition. Conversely, an excessively large speech fundamental frequency variance indicates drastic pitch variations, such as excited speeches, singing, children's speech, and exaggerated animation dubbing. Extreme changes in pitch may alter the formant structure of the sound, thus increasing the difficulty of speech recognition. Here, the fundamental frequency variance of the speech can be estimated from the fundamental frequency sequence of the target audio segment using a fundamental frequency extraction algorithm (e.g., Probabilistic Yin (PYIN), Robust Algorithm for Pitch Tracking (RAPT), etc.), and obtained based on the fundamental frequency sequence; if the fundamental frequency variance of the speech is too small, it can be that the fundamental frequency variance of the speech is less than or equal to a preset low threshold variance, and the preset low threshold variance can be set according to application requirements, and this embodiment of the present disclosure does not limit it; if the fundamental frequency variance of the speech is too large, it can be that the fundamental frequency variance of the speech is greater than or equal to a preset high threshold variance, and the preset high threshold variance can be set according to application requirements, and this embodiment of the present disclosure does not limit it.

[0063] For each target audio segment, the recognition difficulty of the target audio segment is assessed from multiple evaluation dimensions to obtain multiple scores to be fused. These multiple scores are then weighted and summed to obtain the recognition difficulty score of the target audio segment. The weighting allocation used when weighting and summing the multiple scores to obtain the recognition difficulty score of the target audio segment can be set according to application requirements, and this embodiment does not impose any limitations on this.

[0064] In this embodiment of the disclosure, for each target audio segment, if the recognition difficulty score of the target audio segment is in a preset high score range, a large-parameter model is selected from multiple candidate recognition models as the second target model corresponding to the target audio segment; if the recognition difficulty score of the target audio segment is in a preset medium score range, a medium-parameter model is selected from multiple candidate recognition models as the second target model corresponding to the target audio segment; if the recognition difficulty score of the target audio segment is in a preset low score range, a small-parameter model is selected from multiple candidate recognition models as the second target model corresponding to the target audio segment. The large-parameter model can be a Whisper model with a total parameter count of 1550MB; the medium-parameter model can be a Whisper model with a total parameter count of 796MB; and the small-parameter model can be a Whisper model with a total parameter count of 244MB.

[0065] After determining multiple second target models that correspond one-to-one with multiple target audio segments, a concurrent processing mechanism can be used to perform speech recognition on the multiple target audio segments to obtain multiple text segments. In one example, a number of second recognition threads can be started, and the multiple target audio segments can be relatively evenly distributed among the number of second recognition threads. This allows each second recognition thread to call the second target model corresponding to its obtained target audio segment when processing that target audio segment, and perform speech recognition on that target audio segment to obtain the corresponding text segment. The number of second targets can be set according to application requirements, for example, it can be set to 12, and this embodiment of the disclosure does not limit this; the text segments can be in SRT format.

[0066] For example, if there are 240 target audio segments and 12 second recognition threads, the 240 target audio segments can be relatively evenly distributed among the 12 second recognition threads. That is, each second recognition thread obtains 20 target audio segments. Then, for each second recognition thread, the second recognition thread can perform speech recognition on its 20 target audio segments to obtain 20 text segments that correspond one-to-one with these 20 target audio segments.

[0067] Through the above methods, in this embodiment of the disclosure, a recognition difficulty score for each target audio segment can be obtained. For each target audio segment, based on its recognition difficulty score, a second target model corresponding to the target audio segment is determined from multiple candidate recognition models. Then, following a concurrent processing mechanism, multiple second target models corresponding one-to-one with multiple target audio segments are used to perform speech recognition on multiple target audio segments to obtain multiple text segments. In other words, in this embodiment of the disclosure, based on the principle of "using computing resources wisely," more powerful computing resources (e.g., large-parameter models) are used for complex target audio segments to ensure the accuracy of the speech recognition results; while for simple and clear target audio segments, lightweight models (e.g., small-parameter models) are used to improve the speech recognition speed. This fine-grained, adaptive recognition model selection strategy, compared to the "one-size-fits-all" approach of selecting a uniform recognition model for the entire target video, can achieve a higher global average recognition accuracy within the same or even shorter total time, while effectively saving computing resource costs.

[0068] Furthermore, when obtaining the recognition difficulty score for each target audio segment, multiple evaluation dimensions can be determined (e.g., the first dimension corresponding to the signal-to-noise ratio, the second dimension corresponding to the speech speed, and the third dimension corresponding to the speech fundamental frequency variance). For each target audio segment, the recognition difficulty is evaluated from these multiple dimensions to obtain multiple scores to be fused. Then, based on these multiple scores, the recognition difficulty score for the target audio segment is obtained. This overcomes the limitations of a single indicator, comprehensively and accurately quantifying the true recognition difficulty of the target audio segment, laying a more reliable decision-making foundation for the subsequent selection of an adaptive recognition model.

[0069] In some optional implementations, step S104, namely, "translating multiple text segments to obtain target subtitle text," may include: Obtain the video category of the target video; Based on video category, translation prompts are constructed; Using a target large model, and based on translation prompts, multiple text segments are translated to obtain the target subtitle text.

[0070] The video categories may include, but are not limited to, professional academic, legal and judicial, medical and health, complex acoustic environments, important business meetings, dialect communication, academic lectures, business and finance, IT, film, documentary narration, standard meeting minutes, daily dialogue, clear broadcasts, simple teaching, entertainment, and telephone recordings.

[0071] After obtaining the video category of the target video, translation prompts can be constructed based on the video category. In other words, translation prompts can be constructed based on the identification considerations for the video category. For example, when the video category is medical and health, the identification considerations could include: ensuring the absolute accuracy of professional medical terminology, rigorous and objective sentence structure, and conforming to the standard terminology of the medical industry; as another example, when the video category is film, the identification considerations could include: preserving the semantics of the film scene and the colloquial expression style; and as yet another example, when the video category is telephone recording, the identification considerations could include: handling common vague, elliptical, and incomplete sentence structures in the recorded text, transforming fragmented spoken dialogue into fluent written language that conforms to the target language's habits.

[0072] After obtaining the video category of the target video and constructing translation prompts based on the video category, a target large model can be used to translate multiple text segments based on the translation prompts to obtain the target subtitle text. The target large model can be a Large Language Model (LLM). Here, the LLM can be pre-trained or trained, specifically a neural network model (e.g., an autoregressive generative model with a Transformer architecture), which possesses general language knowledge, world knowledge, and domain-specific expertise.

[0073] Through the above methods, in this embodiment of the disclosure, the video category of the target video can be obtained, and translation prompts can be constructed based on the video category. Then, using the target big data model, multiple text segments can be translated based on the translation prompts to obtain the target subtitle text. In other words, in this embodiment of the disclosure, by identifying the video category of the target video, specialized translation prompts can be constructed, upgrading the translation capability of the target big data model from "general" to "domain-specific". This ensures the accurate conversion of professional terms in vertical fields (e.g., law, medicine, technology, etc.) and automatically adapts to the stylistic features of different scenarios (e.g., academic rigor, vivid movie dialogue). This effectively solves the semantic deviation problem of general translation in vertical fields and improves the professionalism, accuracy, and scenario fit of the target subtitle text.

[0074] Furthermore, in this embodiment of the disclosure, when translating multiple text segments based on translation prompts to obtain the target subtitle text, the following can be done: Multiple text fragments are spliced ​​together to obtain the text to be translated; Using a target large model, the text to be translated is processed based on translation prompts to obtain the target subtitle text.

[0075] In this process, when concatenating multiple text segments to obtain the text to be translated, the segments can be concatenated according to the timeline identifier of each target audio segment. This ensures that after using the target large model and translation prompts to translate the text to obtain the target subtitle text, the target subtitle text and audio data are precisely aligned.

[0076] In other words, in this embodiment of the disclosure, multiple text fragments can be concatenated to obtain the text to be translated. Then, the target big data model is used to translate the text based on translation prompts to obtain the target subtitle text. This method integrates all text fragments into a complete text to be translated, providing the target big data model with the richest and most reliable contextual information. This enables the target big data model to translate the text based on a full understanding of word meanings, referential relationships, and logical flow, thereby ensuring consistency in terminology and language style, and greatly improving the accuracy and readability of the target subtitle text.

[0077] In this embodiment of the disclosure, when translating multiple text segments based on translation prompts to obtain the target subtitle text, the following can also be done: Using a large target model, each text segment is translated based on translation prompts to obtain translation results for multiple segments; The translation results of multiple segments are spliced ​​together to obtain the target subtitle text.

[0078] When concatenating the translation results of multiple segments to obtain the target subtitle text, the translation results of multiple segments can be concatenated according to the timeline identifier of each target audio segment. This ensures that the resulting target subtitle text and audio data are precisely aligned.

[0079] In other words, in this embodiment, the target large model can be used first to translate each text segment based on translation prompts to obtain multiple segment translation results. These multiple segment translation results are then concatenated to obtain the target subtitle text. This method translates individual text segments independently, greatly reducing the dependence on the context length of the target large model and making the computational resource requirements of the target large model more controllable. Simultaneously, the failure to translate a single text segment does not affect the global translation result, improving the fault tolerance and stability of the subtitle generation method. This is an efficient and robust engineering implementation scheme.

[0080] Furthermore, the subtitle generation method provided in this disclosure may further include: Identify multiple keyword subtitle contents from the target subtitle text; For each keyword subtitle, determine the first target frame corresponding to the keyword subtitle from multiple frames of the target video; establish the association between the keyword subtitle and the first target frame.

[0081] Among these, multiple keyword subtitles can be anchor subtitles corresponding to key nodes in the target video. Here, key nodes can be, but are not limited to, points of theme transition, points where core questions are raised or answered, points where important viewpoints or conclusions appear, points of summary statements, points where key events occur, etc.

[0082] In this embodiment of the disclosure, a content determination model can be used to determine multiple keyword subtitle contents from the target subtitle text. The content determination model can be an LLM (Limited Linear Model). For example, a content extraction prompt can be constructed, and the content determination model can be used to determine multiple keyword subtitle contents from the target subtitle text based on the content extraction prompt. The content extraction prompt can be: As a professional video content analysis expert, please carefully analyze the target subtitle text and identify anchor subtitle content (e.g., anchor sentences) that represent key nodes in the target video. Key nodes can be, but are not limited to, points of topic transition, points where core questions are raised or answered, points where important viewpoints or conclusions appear, summary statements, and points where key events occur.

[0083] In this embodiment of the disclosure, after determining multiple keyword subtitle contents from the target subtitle text, for each keyword subtitle content, a first target screen corresponding to the keyword subtitle content can be determined from multiple video frames of the target video, and an association relationship between the keyword subtitle content and the first target screen can be established.

[0084] Furthermore, based on the above, the subtitle generation method provided in this disclosure embodiment may further include: In response to a video jump request carrying target subtitle content, a second target frame corresponding to the target subtitle content is determined from multiple video frames; Generate a video jump instruction corresponding to the second target screen; Send the video jump instruction to the terminal device.

[0085] The video jump request can be initiated by the target user via their terminal device, carrying target subtitle content. This target subtitle content can correspond to one of multiple keyword subtitle contents. In one example, the target user can enter at least one target word in the progress bar of the target video to combine at least one target word into the target subtitle content.

[0086] After obtaining the target subtitle content, the cosine similarity between the target subtitle content and multiple keyword subtitle contents can be calculated as the second similarity. The keyword subtitle content with the highest second similarity among the multiple keyword subtitle contents is then taken as the user intent content. Next, a first target frame that is associated with the user intent content is determined from multiple video frames and designated as the second target frame. Subsequently, a video jump instruction corresponding to the second target frame can be generated and sent to the terminal device. The terminal device can then adjust the playback progress of the target video according to the video jump instruction, that is, adjust the current playback frame of the target video to the second target frame.

[0087] Through the above methods, in this embodiment of the disclosure, multiple keyword subtitle contents can be determined from the target subtitle text. For each keyword subtitle content, a first target frame corresponding to the keyword subtitle content is determined from multiple frames of the target video, and an association relationship is established between the keyword subtitle content and the first target frame. Based on this, in this embodiment of the disclosure, in response to a video jump request carrying target subtitle content, a second target frame associated with the target subtitle content can be determined from multiple frames of video, and a video jump instruction corresponding to the second target frame can be generated. The video jump instruction is then sent to the terminal device. In other words, in this embodiment of the disclosure, "semantic-level navigation" of video content can be achieved by intelligently extracting keyword subtitle content and accurately associating it with the corresponding video frame. In this way, users can jump to the relevant video segment with one click by inputting any target subtitle content, completely eliminating the blindness of video jump achieved by dragging a progress bar in the traditional solution, greatly improving the information retrieval and consumption efficiency of the target video, and significantly enhancing the user experience.

[0088] The following, combined with Figure 3 The complete process of a subtitle generation method provided in this disclosure embodiment will be described.

[0089] The communication interface module 301 receives video translation requests sent by the target user based on the terminal device, and determines the target video based on the video translation requests.

[0090] The target video is sent to the offline subtitle management module 302 via the communication interface module 301. The offline subtitle management module 302 performs unified translation scheduling. Specifically, the offline subtitle management module 302 can send the target video to the video subtitle segmentation module 303, so that the video subtitle segmentation module 303 can segment the audio data of the target video to obtain multiple target audio segments, and return the multiple target audio segments to the offline subtitle management module 302. The offline subtitle management module 302 sends the multiple target audio segments to the subtitle recognition module 304, so that the subtitle recognition module 304 can perform speech recognition on the multiple target audio segments according to the concurrent processing mechanism, using the first target model 305 (e.g., Whisper model), to obtain multiple text segments, and return the multiple text segments to the offline subtitle management module 302. The offline subtitle management module 302 concatenates the multiple text segments to obtain the text to be translated, and uses the target large model 306 (e.g., LLM) to translate the text to be translated based on translation prompts to obtain the target subtitle text.

[0091] Subsequently, the offline subtitle management module 302 can merge the target subtitle text into the target video to obtain a target video with target subtitle text, and send the target video with target subtitle text to the terminal device through the communication interface module 301 for the target user to watch.

[0092] Please see Figure 4 This is a schematic diagram illustrating an application scenario of a subtitle generation method provided in this embodiment.

[0093] The subtitle generation method provided in this disclosure is applied to electronic devices. These electronic devices can be servers, workbenches, mainframe computers, or other similar computing devices.

[0094] Here, electronic devices are used for: Identify the target video; The audio data of the target video is segmented to obtain multiple target audio segments; According to the concurrent processing mechanism, speech recognition is performed on multiple target audio segments to obtain multiple text segments; among them, multiple text segments correspond one-to-one with multiple target audio segments; Multiple text segments are translated to obtain the target subtitle text.

[0095] In this embodiment of the disclosure, the target video can be determined in response to a video translation request. The video translation request can be initiated by the target user via a terminal device through the World Wide Web (Web) or a native application (NA), and carries a video identifier. Here, the terminal device can be a conventional computer, smartphone, personal digital processing unit, or other similar computing device.

[0096] It should be noted that, in the embodiments disclosed herein, Figure 4 The application scenario diagrams shown are for illustrative purposes only and are not restrictive. Those skilled in the art can use them as a basis for their own interpretation. Figure 4 The examples may be modified in various obvious ways and / or substitutions, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of this disclosure.

[0097] To better implement the aforementioned subtitle generation method, this disclosure also provides a subtitle generation apparatus, which can be integrated into an electronic device. The electronic device can be a server, workbench, mainframe computer, or other similar computing device. The following will be combined with... Figure 5 The schematic block diagram shown illustrates a subtitle generation apparatus 500 provided in the disclosed embodiment.

[0098] Subtitle generation device 500, including: The video determination unit 501 is used to determine the target video; The audio segmentation unit 502 is used to segment the audio data of the target video to obtain multiple target audio segments; The speech recognition unit 503 is used to perform speech recognition on multiple target audio segments according to a concurrent processing mechanism to obtain multiple text segments; wherein, the multiple text segments correspond one-to-one with the multiple target audio segments. The text translation unit 504 is used to translate multiple text segments to obtain the target subtitle text.

[0099] In some alternative implementations, the audio segmentation unit 502 is used for: The audio data is segmented according to the pause intervals to obtain multiple initial audio segments; The voiceprint features of each initial audio segment are obtained to obtain multiple voiceprint features; Based on multiple voiceprint features, at least some of the initial audio segments from multiple initial audio segments are merged to obtain multiple target audio segments.

[0100] In some alternative implementations, the audio segmentation unit 502 is used for: Based on multiple voiceprint features, adjacent initial audio segments with similar voiceprint features are clustered into an audio segment group to obtain multiple audio segment groups; Each audio segment group is used as a target segment group, and the total duration of the first segment of the initial audio segments included in the target segment group is obtained; If the total duration of the first segment does not exceed the preset duration threshold, the initial audio segments included in the target segment group are merged to obtain the target audio segment. Alternatively, if the total duration of the first segment exceeds a preset duration threshold, the target segment group is split into multiple lower-level audio groups; the initial audio segments included in each lower-level audio group are merged to obtain the target audio segment; wherein the total duration of the second segment of the initial audio segments included in the lower-level audio group does not exceed the preset duration threshold.

[0101] In some alternative implementations, the speech recognition unit 503 is used for: Obtain the recognition difficulty level of the target video; Based on the recognition difficulty level, the first target model is determined from multiple candidate recognition models; Following the concurrent processing mechanism, the first target model is used to perform speech recognition on multiple target audio segments to obtain multiple text segments.

[0102] In some alternative implementations, the speech recognition unit 503 is used for: Obtain a recognition difficulty score for each target audio segment; For each target audio segment, based on the recognition difficulty score of the target audio segment, a second target model corresponding to the target audio segment is determined from multiple candidate recognition models; According to the concurrent processing mechanism, multiple second target models, which correspond one-to-one with multiple target audio segments, are used to perform speech recognition on multiple target audio segments to obtain multiple text segments.

[0103] In some alternative implementations, the speech recognition unit 503 is used for: Define multiple evaluation dimensions; For each target audio segment, the recognition difficulty of the target audio segment is evaluated from multiple evaluation dimensions to obtain multiple scores to be fused; based on the multiple scores to be fused, the recognition difficulty score of the target audio segment is obtained.

[0104] In some alternative implementations, the multiple evaluation dimensions include at least one of a first dimension corresponding to the signal-to-noise ratio, a second dimension corresponding to the speech speed, and a third dimension corresponding to the speech fundamental frequency variance.

[0105] In some alternative implementations, the video determination unit 501 is used for one of the following: In response to the video translation request, the target video is identified; Identify the first candidate video; obtain the original language category of the first candidate video; if the original language category is different from the reference language category, use the first candidate video as the target video.

[0106] In some optional implementations, the text translation unit 504 is used for: Obtain the video category of the target video; Based on video category, translation prompts are constructed; Using a target large model, and based on translation prompts, multiple text segments are translated to obtain the target subtitle text.

[0107] In some optional implementations, the text translation unit 504 is used for: Multiple text fragments are spliced ​​together to obtain the text to be translated; Using a target large model, the text to be translated is processed based on translation prompts to obtain the target subtitle text.

[0108] In some optional implementations, the text translation unit 504 is used for: Using a large target model, each text segment is translated based on translation prompts to obtain translation results for multiple segments; The translation results of multiple segments are spliced ​​together to obtain the target subtitle text.

[0109] In some alternative implementations, the subtitle generation apparatus 500 further includes an index association unit for: Identify multiple keyword subtitle contents from the target subtitle text; For each keyword subtitle, determine the first target frame corresponding to the keyword subtitle from multiple frames of the target video; establish the association between the keyword subtitle and the first target frame.

[0110] In some alternative implementations, the subtitle generation apparatus 500 further includes an index response unit for: In response to a video jump request carrying target subtitle content, a second target frame that is associated with the target subtitle content is determined from multiple video frames; wherein the target subtitle content corresponds to one of multiple keyword subtitle contents. Generate a video jump instruction corresponding to the second target screen; The video jump instruction is sent to the terminal device; the terminal device is used to adjust the playback progress of the target video according to the video jump instruction.

[0111] In this embodiment of the disclosure, the specific functions and examples of each unit in the subtitle generation device 500 can be found in the relevant descriptions of the corresponding steps in the aforementioned subtitle generation method embodiments, and will not be repeated here.

[0112] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0113] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0114] Figure 6 A schematic structural block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Electronic device 600 is intended to represent various forms of digital computers, such as in-vehicle computing devices, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 600 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0115] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0116] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of renderers, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0117] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the caption generation method. For example, in some embodiments, the caption generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the caption generation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured as a subtitle generation method by any other suitable means (e.g., by means of firmware).

[0118] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data optimization device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a rendering device (e.g., a cathode ray tube (CRT) renderer or a liquid crystal display (LCD)) for rendering information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices are also used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0122] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0123] A computer system can include client and server components. Clients and servers are generally located far apart and typically interact via a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server in a distributed system, or a server incorporating blockchain technology.

[0124] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute a subtitle generation method.

[0125] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements a subtitle generation method.

[0126] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure is achieved, and this is not limited herein. Furthermore, in this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Additionally, "multiple" in this disclosure can be understood as at least two.

[0127] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating subtitles, comprising: Identify the target video; The audio data of the target video is segmented to obtain multiple target audio segments; According to the concurrent processing mechanism, speech recognition is performed on the multiple target audio segments to obtain multiple text segments; wherein, the multiple text segments correspond one-to-one with the multiple target audio segments; The multiple text segments are translated to obtain the target subtitle text.

2. The method according to claim 1, wherein, The segmentation of the audio data of the target video to obtain multiple target audio segments includes: The audio data is segmented according to the pause intervals to obtain multiple initial audio segments; The voiceprint features of each initial audio segment are obtained to obtain multiple voiceprint features; Based on the multiple voiceprint features, at least some of the initial audio segments are merged to obtain the multiple target audio segments.

3. The method according to claim 2, wherein, The step of merging at least a portion of the initial audio segments from the plurality of initial audio segments based on the plurality of voiceprint features to obtain the plurality of target audio segments includes: Based on the multiple voiceprint features, adjacent initial audio segments with similar voiceprint features are clustered into an audio segment group to obtain multiple audio segment groups; Each audio segment group is taken as a target segment group, and the total duration of the first segment of the initial audio segments included in the target segment group is obtained; If the total duration of the first segment does not exceed a preset duration threshold, the initial audio segments included in the target segment group are merged to obtain the target audio segment. Alternatively, if the total duration of the first segment exceeds a preset duration threshold, the target segment group is split into multiple lower-level audio groups; the initial audio segments included in each lower-level audio group are merged to obtain the target audio segment; wherein the total duration of the second segment of the initial audio segments included in the lower-level audio group does not exceed the preset duration threshold.

4. The method according to claim 2, wherein, The method involves performing speech recognition on the multiple target audio segments according to a concurrent processing mechanism to obtain multiple text segments, including: Obtain the recognition difficulty level of the target video; Based on the recognition difficulty level, a first target model is determined from multiple candidate recognition models; According to the concurrent processing mechanism, the first target model is used to perform speech recognition on the multiple target audio segments to obtain the multiple text segments.

5. The method according to claim 2, wherein, The method involves performing speech recognition on the multiple target audio segments according to a concurrent processing mechanism to obtain multiple text segments, including: Obtain a recognition difficulty score for each of the target audio segments; For each target audio segment, based on the recognition difficulty score of the target audio segment, a second target model corresponding to the target audio segment is determined from multiple candidate recognition models; According to the concurrent processing mechanism, multiple second target models corresponding one-to-one with the multiple target audio segments are used to perform speech recognition on the multiple target audio segments to obtain the multiple text segments.

6. The method according to claim 5, wherein, The process of obtaining the recognition difficulty score for each target audio segment includes: Define multiple evaluation dimensions; For each target audio segment, the recognition difficulty of the target audio segment is evaluated from the multiple evaluation dimensions to obtain multiple scores to be fused; based on the multiple scores to be fused, the recognition difficulty score of the target audio segment is obtained.

7. The method according to claim 6, wherein, The plurality of evaluation dimensions include at least one of a first dimension corresponding to signal-to-noise ratio, a second dimension corresponding to speech speed, and a third dimension corresponding to speech fundamental frequency variance.

8. The method according to claim 1, wherein, The determination of the target video includes one of the following: In response to a video translation request, the target video is determined; A first candidate video is determined; the original language category of the first candidate video is obtained; if the original language category is different from the reference language category, the first candidate video is used as the target video.

9. The method according to claim 1, wherein, The process of translating the multiple text segments to obtain the target subtitle text includes: Obtain the video category of the target video; Based on the video category, translation prompts are constructed; Using the target large model, based on the translation prompt words, the multiple text segments are translated to obtain the target subtitle text.

10. The method according to claim 9, wherein, The process of using a target large model to translate multiple text segments based on the translation prompts to obtain the target subtitle text includes: The multiple text fragments are concatenated to obtain the text to be translated; Using the target large model and based on the translation prompts, the text to be translated is translated to obtain the target subtitle text.

11. The method according to claim 9, wherein, The process of using a target large model to translate multiple text segments based on the translation prompts to obtain the target subtitle text includes: Using the target large model, and based on the translation prompt words, each of the text segments is translated to obtain multiple segment translation results; The translation results of the multiple segments are concatenated to obtain the target subtitle text.

12. The method according to any one of claims 1 to 11, further comprising: Multiple keyword subtitle contents are determined from the target subtitle text; For each keyword screen content, a first target screen corresponding to the keyword screen content is determined from multiple video frames of the target video; Establish the association between the keyword screen content and the first target screen.

13. The method of claim 12, further comprising: In response to a video jump request carrying target subtitle content, a second target screen corresponding to the target subtitle content is determined from the multiple video frames; wherein the target subtitle content corresponds to one of the multiple keyword subtitle contents; Generate a video jump instruction corresponding to the second target screen; The video jump instruction is sent to the terminal device; wherein the terminal device is used to adjust the playback progress of the target video according to the video jump instruction.

14. A subtitle generation device, comprising: The video determination unit is used to determine the target video; An audio segmentation unit is used to segment the audio data of the target video to obtain multiple target audio segments; The speech recognition unit is used to perform speech recognition on the plurality of target audio segments according to a concurrent processing mechanism to obtain a plurality of text segments; wherein the plurality of text segments correspond one-to-one with the plurality of target audio segments; The text translation unit is used to translate the multiple text segments to obtain the target subtitle text.

15. An electronic device comprising: At least one processor; A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program; wherein, When the computer program is executed by a processor, it can implement the method of any one of claims 1 to 13.

Images