Method and device for automatic translation of subtitles based on large models, and storage medium

Abstract

The present application relates to the technical field of video data processing, and particularly relates to a subtitle automatic translation method and device based on a large model and a storage medium. The method comprises the following steps: obtaining subtitle data and video data and preprocessing to obtain a time axis index table; constructing a semantic model based on the time axis index table to generate a semantic vector; performing feature extraction on the video data to obtain a speech feature vector and a video feature vector, and performing feature fusion on the semantic vector, the speech feature vector and the video feature vector to construct a fused feature vector; constructing a professional term library and generating a preliminary translation result according to the fused feature vector; constructing a subtitle evaluation model according to the preliminary translation result to obtain an adapted subtitle. The present application realizes accurate translation of subtitle data in a video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video data processing, and particularly relates to a subtitle automatic translation method and device based on a large model and a storage medium. BACKGROUND

[0002] With the acceleration of globalization and the widespread dissemination of multimedia content, the demand for cross-language video subtitles is growing. Traditional subtitle translation methods rely on manual processing, which is inefficient and costly, and cannot meet the real-time and large-scale translation needs. In recent years, the rapid development of artificial intelligence and natural language processing technology, especially the emergence of large language models (LLM), has provided a new technical path for automated and intelligent subtitle translation, promoting the transformation from manual to intelligent in this field.

[0003] Existing subtitle translation techniques mainly rely on rule matching or traditional machine translation models, lacking deep understanding of context semantics, and being difficult to handle complex situations such as multilingual mixing, professional terminology, and cultural differences. At the same time, existing methods often ignore the multi-modal information of the video (such as audio, picture content), resulting in translation results that are out of context with the video, weak time axis synchronization capability, inconsistent translation style, and overall results that are not natural and accurate. SUMMARY

[0004] The present application aims to provide a subtitle automatic translation method and device based on a large model and a storage medium to solve at least one of the problems in the prior art.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A subtitle automatic translation method based on a large model, comprising:

[0007] Obtaining subtitle data and video data and preprocessing to obtain a time axis index table;

[0008] Building a semantic model based on the time axis index table to generate a semantic vector;

[0009] Extracting features from the video data to obtain a speech feature vector and a video feature vector, and performing feature fusion on the semantic vector, the speech feature vector, and the video feature vector to construct a fused feature vector;

[0010] Building a professional term library and generating a preliminary translation result based on the fused feature vector;

[0011] Building a subtitle evaluation model based on the preliminary translation result to obtain an adapted subtitle.

[0012] Preferably, the start time and the end time corresponding to each caption in the caption data are extracted, and the captions are corresponded to the start time and the end time, so as to establish a time axis index table.

[0013] Preferably, the size of the sliding window is set to 3 captions, the sliding step is set to 1 caption each time, the captions in the window are segmented to obtain caption words, the word frequency of the caption words is calculated, the ratio of the number of occurrences of each caption word in the captions in the window to the total number of caption words in the captions in the window is taken as the word frequency, the word frequencies of the caption words in the window are arranged in descending order, and the top 10 word frequencies are taken as the word frequency features.

[0014] The proportion of sentence types of the captions in the window is extracted, the proportion of sentence types is the proportion of the captions being declarative sentences, interrogative sentences and exclamatory sentences in the window, the average number of words of the captions in the window is calculated, the average number of words is the ratio of the total number of caption words of the captions in the window to the number of sentences of the captions in the window, the number of caption words identical to the preset conjunction words in the captions in the window is counted as the number of window conjunction words, the number of caption words identical to the preset pronouns in the captions in the window is counted as the number of window pronouns, and the proportion of sentence types, the average number of words, the number of window conjunction words and the number of window pronouns are taken as the syntactic features.

[0015] The semantic vector is generated according to the word frequency features and the syntactic features, and the expression of the semantic vector is C(i) = 0.6xFtf(i) + 0.4xFsy(i), wherein C(i) represents the semantic vector, i represents the caption sentence number, Ftf(i) represents the word frequency features, and Fsy(i) represents the syntactic features.

[0016] The context consistency check is performed according to the semantic vector, so as to trigger the semantic vector regeneration mechanism.

[0017] Preferably, the speech feature vector of the audio signal corresponding to each caption is extracted according to the audio signal, the frame length of the audio signal is set to 25 ms, the frame shift is set to 10 ms, and the mel-frequency cepstral coefficients 2-13 of the audio signal are reserved as the speech feature vector.

[0018] The video feature vector of the video image corresponding to each caption is extracted according to the video image, the color histogram of the video image is extracted, the change rate of the proportion of different colors in the color histograms of the video images of adjacent frames is taken as the video color feature, and the number of faces in the video image is counted as the video face number, and the vector composed of the video color feature and the video face number is taken as the video feature vector.

[0019] The feature weight is constructed according to a video scene, and a fusion feature vector is constructed according to the feature weight, a semantic vector, a speech feature vector and a video feature vector; the semantic vector, the speech feature vector and the video feature vector are reduced to the same dimension by principal component analysis, and the fusion feature vector is analyzed.

[0020] Preferably, a hierarchical term table is constructed, which comprises source terms, target terms and domain labels, the source terms are term expressions in a source language, the target terms are translated term expressions in a target language corresponding to the source terms, and the domain labels are domains to which the terms belong;

[0021] A distance score is determined according to the subtitle word and the professional term library, and an expression of the distance score is Se(u,v)=1-ED(u,v) / max(len(u),len(v)), wherein Se(u,v) represents the distance score, ED(u,v) represents an edit distance between the subtitle word and a source term in the professional term library, len(u) represents a character length of the subtitle word, len(v) represents a character length of the source term, u represents the subtitle word, and v represents the source term;

[0022] When the distance score is greater than or equal to 0.85, a semantic score is determined according to the subtitle word and the professional term library, a synonym set of the subtitle word in the professional term library is extracted, and a Jaccard similarity coefficient of the synonym set is taken as the semantic score;

[0023] A domain label corresponding to the subtitle word in each sentence of the subtitle is extracted respectively to perform domain matching judgment, if the domain label corresponding to the subtitle word of the current subtitle completely matches the domain labels corresponding to the subtitle words of the previous and next subtitles, the domain score is set to 1, if the domain label corresponding to the subtitle word of the current subtitle is partially related to the domain labels corresponding to the subtitle words of the previous and next subtitles, the domain score is set to 0.5, and if the domain label corresponding to the subtitle word of the current subtitle does not match the domain labels corresponding to the subtitle words of the previous and next subtitles, the domain score is set to 0.2.

[0024] Preferably, a matching score is analyzed based on the distance score, the semantic score and the domain score, and an expression of the matching score is St(u,v)=0.4×Se(u,v)+0.3×Sj(u,v)+0.3×Sd, wherein St(u,v) represents the matching score, Sj(u,v) represents the semantic score, and Sd represents the domain score.

[0025] Preferably, a BLEU score of the preliminary translation result is taken as the quality score;

[0026] The translated adaptive subtitle is taken as a historical adaptive subtitle, and style feature extraction is performed on the historical adaptive subtitle, the average number of words of each subtitle of the historical adaptive subtitle, the 100 highest-frequency subtitle words in the historical adaptive subtitle, and the proportion of the sentence types of the subtitles of the historical adaptive subtitle are taken as historical style features, the average number of words of the subtitles in a window, the top 10 highest-frequency subtitle words in the window, and the proportion of the sentence types of the subtitles in the window are taken as subtitle style features, and the cosine similarity of the historical style features and the subtitle style features is taken as a style similarity;

[0027] The preliminary translation result is optimized based on the quality score and the style similarity, when the quality score is less than 0.7, the expression of the semantic vector is updated as C(i) = 0.5 x Ftf(i) + 0.5 x Fsy(i), and re-translation is triggered, when the style similarity is less than 0.8, the preliminary translation result is translated back to the source language according to the professional term library and re-translation is triggered, and when the quality score is greater than or equal to 0.7 and the style similarity is greater than or equal to 0.8, the preliminary translation result is taken as a final translation result.

[0028] Preferably, the display time of the translation is analyzed according to the final translation result, and the expression of the display time of the translation is Tn = To x (Lt / Ls) x (Rs / Rt), wherein Tn represents the display time of the translation, To represents the display time of the original subtitle, Lt represents the text length of the subtitle, Ls represents the text length of the final translation result, Rs represents a preset source language reading speed, and Rt represents a preset target language reading speed.

[0029] The length of the subtitle is revised according to the text length of the subtitle and the text length of the final translation result, if Lt > 1.5 x Ls, a translation length compression strategy is triggered, if Lt < 0.7 x Ls, a translation length expansion strategy is triggered, and the display time of the final translation result is set as the display time of the translation to obtain an adaptive subtitle.

[0030] In another aspect, the present application also provides a subtitle automatic translation device based on a large model, comprising:

[0031] A data acquisition unit is configured to acquire subtitle data and video data and pre-process the subtitle data and the video data to obtain a time axis index table;

[0032] A semantic analysis unit is configured to construct a semantic model based on the time axis index table to generate a semantic vector;

[0033] A feature fusion unit is configured to extract features from the video data to obtain a speech feature vector and a video feature vector, and to fuse the semantic vector, the speech feature vector, and the video feature vector to construct a fused feature vector;

[0034] The subtitle translation unit is configured to build a professional term library and generate a preliminary translation result according to the fusion feature vector.

[0035] The translation evaluation unit is configured to build a subtitle evaluation model according to the preliminary translation result to obtain an adapted subtitle.

[0036] In another aspect, the present application also provides a storage medium storing instructions which, when executed on a computer, cause the computer to perform the large model-based automatic subtitle translation method described above.

[0037] The present application has the following advantages: by introducing a large language model and a multi-modal fusion mechanism, the full-process intelligence of subtitle translation from text processing to context understanding is realized, the system has strong semantic understanding ability, professional term recognition ability, multi-language support ability and style consistency control ability, the accuracy, naturalness and practicality of translation are improved, at the same time, the automatic processing process greatly reduces the labor cost and time consumption, and is suitable for real-time and batch translation requirements of various video contents. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 The flowchart of the large model-based automatic subtitle translation method of the present embodiment.

[0040] Figure 2 The flowchart of the generation method of the preliminary translation result of the present embodiment.

[0041] Figure 3 The flowchart of the construction method of the subtitle evaluation model of the present embodiment.

[0042] Figure 4 The structural schematic diagram of the large model-based automatic subtitle translation device of the present embodiment. DETAILED DESCRIPTION

[0043] The large model-based subtitle automatic translation method, device and storage medium disclosed by the present application are further described in detail below in combination with the drawings and specific embodiments. It should be noted that the technical features described in the following embodiments or combinations of technical features should not be considered in isolation, and they can be combined with each other to achieve better technical effects. In the drawings of the following embodiments, the same reference numerals appearing in each drawing represent the same features or components and can be applied to different embodiments. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0044] It should be noted that the structures, proportions, sizes, etc. shown in the drawings attached to the present specification are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read, and are not used to limit the conditions for implementing the application. Any modification of structure, change of proportion relationship or adjustment of size, which does not affect the effect and purpose that the application can produce, should fall within the scope of the technical content disclosed by the application. The scope of the preferred embodiments of the present application includes additional implementations, in which the functions can be performed in a substantially simultaneous manner or in reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0045] Techniques, methods, and equipment known to those of ordinary skill in the relevant art can not be discussed in detail, but in appropriate cases, the techniques, methods, and equipment should be considered as part of the authorized specification. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, not as a limitation. Therefore, other examples of exemplary embodiments can have different values.

[0046] In the description of the embodiments of the present application, " / " means or, "and / or" is used to describe the association between the associated objects, which means that there can be three relationships, such as "A and / or B" means that A and B exist alone, B exists alone, and A and B exist simultaneously. In the description of the embodiments of the present application, "multiple" means two or more embodiments.

[0047] Please refer to Figure 1 As shown in the drawing, the large model-based subtitle automatic translation method of the present embodiment comprises:

[0048] Step S1, obtain subtitle data and video data and preprocess to obtain a time axis index table, the subtitle data is a subtitle file in SRT, ASS, VTT or the like, the video data includes an audio signal, a video image and a video scene, the video scene includes a dialogue, an action and a silence, etc., the video scene is a scene content corresponding to a picture in a video, the dialogue represents a picture in a video for a dialogue between characters, the action includes a fight scene between characters and a running scene of characters, etc. about a high motion intensity scene of characters, and the silence represents a silent part of the corresponding video. Specifically, in step S1, the text cleaning is performed on the subtitle data to remove the same content as the preset useless word in the subtitle data, and the preset useless word includes but is not limited to HTML tags, special characters, format markers and other irrelevant content to the subtitle content.

[0049] Specifically, in step S1, the start time and the end time corresponding to each sentence of the subtitle data are extracted, and the subtitle and the corresponding start time and end time are corresponded to establish a time axis index table.

[0050] Specifically, in step S1, the irrelevant characters and format noise are removed by cleaning and time axis analysis of the subtitle file to establish an accurate time axis index table, which provides a structured input basis for subsequent semantic analysis and multi-modal fusion, and improves the accuracy and efficiency of subsequent processing.

[0051] Please continue to refer to Figure 1 As shown, the subtitle automatic translation method based on a large model further includes:

[0052] Step S2, constructing a semantic model based on the time axis index table to generate a semantic vector.

[0053] Specifically, in step S2, the size of the sliding window is set to 3 sentences of subtitles, and the sliding step is 1 sentence of subtitles each time. The semantic coding analysis is performed on the subtitles in the window to generate a semantic vector.

[0054] Specifically, in step S2, the subtitles in the window are processed by word segmentation to obtain subtitle words, and the word frequency of the subtitle words is calculated. The ratio of the number of occurrences of each subtitle word in the subtitles in the window to the total number of subtitle words in the subtitles in the window is taken as the word frequency. The word frequencies of the subtitle words in the window are arranged in descending order, and the top 10 word frequencies are taken as the word frequency features.

[0055] Specifically, in step S2, the proportion of sentence types of the subtitles in the extraction window is extracted, the proportion of the subtitles being declarative sentences, interrogative sentences and exclamatory sentences, and the average number of words in the subtitles in the window is calculated, the average number of words being the ratio of the total number of subtitle words in the subtitles in the window to the number of sentences in the subtitles in the window, the number of subtitle words in the subtitles in the window that are the same as the preset conjunctions is counted as the number of window conjunctions, the number of subtitle words in the subtitles in the window that are the same as the preset pronouns is counted as the number of window pronouns, and the proportion of sentence types, the average number of words, the number of window conjunctions and the number of window pronouns are taken as syntactic features. The preset conjunctions include but are not limited to conjunctions such as “and”, “but”, “because” and the like, and include conjunctions in all translation languages, such as the English conjunctions “and”, “but”, “because” and the like, and the preset pronouns include but are not limited to pronouns such as “it”, “this”, “that” and the like, and include pronouns in all translation languages, such as the English pronouns “it”, “this”, “that” and the like.

[0056] Specifically, in step S2, a semantic vector is generated according to the word frequency feature and the syntactic feature, and the expression of the semantic vector is: C(i) = 0.6xFtf(i) + 0.4xFsy(i), where C(i) represents the semantic vector, i represents the subtitle sentence number, Ftf(i) represents the word frequency feature, and Fsy(i) represents the syntactic feature. The subtitle sentence number is defined to distinguish the numbers of different sentences in the subtitle data.

[0057] Specifically, in step S2, context consistency checking is performed according to the semantic vector to trigger a semantic vector regeneration mechanism, and context similarity is analyzed according to the semantic vector, the cosine similarity between the semantic vector of the current analysis subtitle and the semantic vector of the last sentence subtitle is taken as the context similarity, if the context similarity is greater than or equal to 0.7, the current semantic vector is retained, if the context similarity is less than 0.7 and greater than or equal to 0.5, the size of the sliding window is enlarged by 2 sentences, and the semantic vector is regenerated, and if the context similarity is less than 0.5, the size of the sliding window is set to 1 sentence, and the semantic vector is regenerated.

[0058] Specifically, in step S2, the local and global semantic features of the subtitles are captured through the sliding window mechanism and the context semantic analysis, the perception ability of the model to the continuity of the dialogue and the change of the context is enhanced, and thus the semantic consistency and naturalness of the translation are improved.

[0059] Please continue to refer to Figure 1 As shown in the figure, the subtitle automatic translation method based on a large model further comprises:

[0060] In step S3, the semantic vector and the video data are fused to construct a fused feature vector.

[0061] Specifically, in step S3, the speech feature vector of the audio signal corresponding to each caption is extracted according to the audio signal, the frame length of the audio signal is set to 25 ms, the frame shift is set to 10 ms, and the mel-frequency cepstral coefficients 2-13 of the audio signal are reserved as the speech feature vector.

[0062] Specifically, in step S3, the video feature vector of the video image corresponding to each caption is extracted according to the video image, the color histogram of the video image is extracted, the change rate of the proportion of different colors in the color histogram of the video image of adjacent frames is taken as the video color feature, and the number of faces in the video image is taken as the video face number, and the vector composed of the video color feature and the video face number is taken as the video feature vector.

[0063] Specifically, in step S3, the feature weight is constructed according to the video scene, the feature weight includes the audio weight, the video weight and the caption weight, when the video scene is dialogue, the audio weight is set to 0.6, the video weight is set to 0.2, and the caption weight is set to 0.2, when the video scene is action, the audio weight is set to 0.3, the video weight is set to 0.6, and the caption weight is set to 0.1, when the video scene is silence, the audio weight is set to 0.1, the video weight is set to 0.7, and the caption weight is set to 0.2.

[0064] Specifically, in step S3, the fusion feature vector is constructed according to the feature weight, the semantic vector, the speech feature vector and the video feature vector, the semantic vector, the speech feature vector and the video feature vector are reduced to the same dimension by principal component analysis, and the fusion feature vector is analyzed, the expression of the fusion feature vector is: H(i) = w1·A(i) + w2·V(i) + w3·C(i), wherein H(i) represents the fusion feature vector, w1 represents the audio weight, w2 represents the video weight, w3 represents the caption weight, A(i) represents the speech feature vector, and V(i) represents the video feature vector.

[0065] Specifically, in the embodiment, the semantic vector, the speech feature vector and the video feature vector are reduced to 20 dimensions, and the embodiment does not specifically limit the dimension of the reduced vector, and the vector is reduced to ensure alignment of the vector, so as to facilitate weighted fusion.

[0066] Specifically, in step S3, the multi-modal information of text, audio and video is fused, and a dynamic weight distribution mechanism is used, so that the translation result is more consistent with the actual context of the video, and the accuracy and scene adaptability of translation are enhanced, which is especially suitable for complex scenes such as dialogue and action.

[0067] Please continue to refer to Figure 1As shown, the large model-based subtitle automatic translation method further comprises:

[0068] Step S4, constructing a professional term library and generating a preliminary translation result according to the fusion feature vector.

[0069] Please refer to Figure 2 As shown, it is a method for generating a preliminary translation result, comprising:

[0070] Step S41, constructing a professional term library.

[0071] Specifically, in the step S41 in the embodiment, a hierarchical term table is constructed, which contains source terms, target terms and field labels. The source term is the term expression in the source language, the target term is the translated term expression in the target language corresponding to the source term, and the field label is the field to which the term belongs. For example, if the source term is "cel l" and the target term is "cell", the field label is "biology". There may also be a source term "cel l" and a target term "battery", and the field label is "electricity", etc. There may be polysemy of a word.

[0072] Please continue to refer to Figure 2 As shown, the method for generating a preliminary translation result further comprises:

[0073] Step S42, determining the distance score according to the subtitle word and the professional term library.

[0074] Specifically, in the step S42 of the embodiment, the distance score is determined according to the subtitle word and the professional term library. The expression of the distance score is Se(u,v) = 1-ED(u,v) / max(len(u),len(v)), where Se(u,v) represents the distance score, ED(u,v) represents the edit distance between the subtitle word and the source term in the professional term library, len(u) represents the character length of the subtitle word, len(v) represents the character length of the source term, u represents the subtitle word, and v represents the source term.

[0075] Please continue to refer to Figure 2 As shown, the method for generating a preliminary translation result further comprises:

[0076] Step S43, determining the semantic score according to the distance score, the subtitle word and the professional term library.

[0077] Specifically, in step S43 of this embodiment, when the distance score is greater than or equal to 0.85, the semantic score is determined based on the subtitle words and the terminology database. The synonym set of the subtitle words and the terminology database is extracted, and the Jaccard similarity coefficient of the synonym set is used as the semantic score. If the synonym set is WordNet, and the subtitle word is "GPU", its overlap with the synonym of "graphics processing unit" in the terminology database is high, and its semantic score is close to 1.

[0078] Please continue reading. Figure 2 As shown, the method for generating the preliminary translation results further includes:

[0079] Step S44: Determine the domain score based on the subtitle words and the professional terminology database.

[0080] Specifically, in step S44 of this embodiment, the domain tags corresponding to the subtitle words in each subtitle are extracted for domain matching judgment. If the domain tag corresponding to the subtitle word of the current subtitle completely matches the domain tags corresponding to the subtitle words of the previous and next subtitles, the domain score is set to 1. If the domain tag corresponding to the subtitle word of the current subtitle is partially related to the domain tags corresponding to the subtitle words of the previous and next subtitles, the domain score is set to 0.5. If the domain tag corresponding to the subtitle word of the current subtitle does not match the domain tags corresponding to the subtitle words of the previous and next subtitles, the domain score is set to 0.2.

[0081] Please continue reading. Figure 2 As shown, the method for generating the preliminary translation results further includes:

[0082] Step S45: Analyze the matching score based on distance score, semantic score, and domain score.

[0083] Specifically, in step S45 of this embodiment, the matching score is analyzed based on the distance score, semantic score, and domain score. The expression for the matching score is: St(u,v)=0.4×Se(u,v)+0.3×Sj(u,v)+0.3×Sd, where St(u,v) represents the matching score, Sj(u,v) represents the semantic score, and Sd represents the domain score.

[0084] Please continue reading. Figure 2 As shown, the method for generating the preliminary translation results further includes:

[0085] Step S46: Generate preliminary translation results based on the matching score and fused feature vector.

[0086] Specifically, in step S46, when the matching score is greater than or equal to 0.8, the source term corresponding to the matching score is replaced in the subtitle data to generate a replaced subtitle, the replaced subtitle is analyzed using a semantic model, and a feature is fused with the video data to construct a replaced feature vector, a cosine similarity of the fused feature vector and the replaced feature vector is taken as a context score, a target term corresponding to the source term with the highest context score is taken as a translation content, and the translation content of the subtitle is taken as a preliminary translation result. The analysis process of the replaced feature vector is the same as that of the fused feature vector, which will not be described in detail in this embodiment.

[0087] Specifically, in step S4, a hierarchical term library is constructed, and a multi-modal feature is combined to intelligently identify and accurately translate professional terms, avoid errors caused by polysemy, and improve the translation quality and reliability of professional field content.

[0088] Please continue to refer to Figure 1 As shown in the figure, the subtitle automatic translation method based on a large model further includes:

[0089] Step S5, constructing a subtitle evaluation model according to the preliminary translation result to obtain an adapted subtitle.

[0090] Please refer to Figure 3 As shown in the figure, it is a construction method of a subtitle evaluation model, which includes:

[0091] Step S51, quality evaluation is performed on the preliminary translation result to obtain a quality score.

[0092] Specifically, in step S51, the BLEU score of the preliminary translation result is taken as the quality score to judge the n-gram matching degree of the preliminary translation result.

[0093] Please continue to refer to Figure 3 As shown in the figure, the construction method of the subtitle evaluation model further includes:

[0094] Step S52, constructing a style model based on historical adapted subtitles to obtain a style similarity.

[0095] Specifically, in step S52, the translated adapted subtitles are taken as historical adapted subtitles, and style features are extracted from the historical adapted subtitles, the average number of words of each subtitle of the historical adapted subtitles, the top 100 subtitle words with the highest frequency in the historical adapted subtitles, and the proportion of sentence types of the subtitles of the historical adapted subtitles are taken as historical style features, the average number of words of the subtitles in a window, the top 10 subtitle words with the highest frequency in the window, and the proportion of sentence types of the subtitles in the window are taken as subtitle style features, and a cosine similarity of the historical style features and the subtitle style features is taken as the style similarity.

[0096] Please continue to refer to Figure 3 As shown in the figure, the method for constructing the subtitle evaluation model further comprises:

[0097] Step S53, the preliminary translation result is optimized based on the quality score and the style similarity to obtain the final translation result.

[0098] Specifically, in step S53, the preliminary translation result is optimized based on the quality score and the style similarity. When the quality score is less than 0.7, the expression of the semantic vector is updated as C(i) = 0.5xFtf(i) + 0.5xFsy(i), and re-translation is triggered. When the style similarity is less than 0.8, the preliminary translation result is translated back to the source language according to the professional term library and re-translation is triggered. When the quality score is greater than or equal to 0.7 and the style similarity is greater than or equal to 0.8, the preliminary translation result is taken as the final translation result.

[0099] Please continue to refer to Figure 3 As shown in the figure, the method for constructing the subtitle evaluation model further comprises:

[0100] Step S54, the translation display time is analyzed according to the final translation result, and a subtitle length revision decision is triggered to obtain an adapted subtitle.

[0101] Specifically, in step S54, the translation display time is analyzed according to the final translation result. The expression of the translation display time is Tn = To x (Lt / Ls) x (Rs / Rt), where Tn represents the translation display time, To represents the original subtitle display time, Lt represents the text length of the subtitle, Ls represents the text length of the final translation result, Rs represents the preset source language reading speed, and Rt represents the preset target language reading speed. The text length is the number of characters.

[0102] Specifically, in step S54, the subtitle length revision decision is triggered according to the text length of the subtitle and the text length of the final translation result. If Lt > 1.5 x Ls, the translation length compression strategy is triggered, the words that can be represented by abbreviations in the final translation result are represented by abbreviations, and the redundant modifiers in the final translation result are omitted. If Lt < 0.7 x Ls, the translation length expansion strategy is triggered, and explanatory phrases are added in the final translation result.

[0103] Specifically, in step S54, the display time of the final translation result is set as the translation display time to obtain the adapted subtitle.

[0104] Specifically, in step S5, the quality evaluation and style consistency control are performed to ensure that the translation result conforms to the language specification and maintains the overall style consistency, and the length adaptation and time axis synchronization mechanism are used to make the display time length and reading experience of the translated subtitle optimal.

[0105] Referring to Figure 4 As shown in the figure, the subtitle automatic translation device based on a large model according to the embodiment comprises:

[0106] A data acquisition unit is configured to acquire and preprocess subtitle data and video data to obtain a time axis index table;

[0107] A semantic analysis unit is configured to construct a semantic model based on the time axis index table to generate a semantic vector;

[0108] A feature fusion unit is configured to extract features from the video data to obtain a speech feature vector and a video feature vector, and perform feature fusion on the semantic vector, the speech feature vector and the video feature vector to construct a fused feature vector;

[0109] A subtitle translation unit is configured to construct a professional term library and generate a preliminary translation result according to the fused feature vector;

[0110] A translation evaluation unit is configured to construct a subtitle evaluation model according to the preliminary translation result to obtain an adapted subtitle.

[0111] The embodiment of the present application also provides a computer readable storage medium storing instructions, when the instructions are run on a computer, the computer performs the subtitle automatic translation method based on a large model as described in the above method embodiment.

[0112] Those of ordinary skill in the art will appreciate that all or certain steps, systems of the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Certain physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable programs, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it is common in the art to use the term computer program, which can include a piece of software, a program module, or a program that runs on a computer, or a combination of these. In addition, it is common in the art to use the term computer readable program, which can include a piece of software, a program module, or a program that runs on a computer, or a combination of these.

[0113] In the above description, the disclosure of the application is not intended to be limited to the aspects described. Rather, the components can be selectively combined in any number of ways, as operative and optional, within the scope of the intended protection of the disclosure. In addition, terms like "include", "comprise", and "have" should be construed as inclusive or open-ended, rather than exclusive or closed, unless specifically defined as such. All technical, scientific or other terms have the meaning understood by those of ordinary skill in the art, unless specifically defined otherwise. Common terms found in dictionaries should not be interpreted too ideally or too unrealistically in the context of the relevant technical documents, unless specifically defined as such by the disclosure. Any changes or modifications made by those of ordinary skill in the art based on the above disclosure are within the scope of protection of the claims.

Claims

1. A method for automatic subtitle translation based on a large model, characterized in that, include: Acquire and preprocess subtitle and video data to obtain a timeline index table; A semantic model is built based on a timeline index table to generate semantic vectors; Set the sliding window size to 3 lines of subtitles, with each sliding step being 1 line of subtitles. Perform word segmentation on the subtitles in the window to obtain subtitle words, and calculate the word frequency of the subtitle words. The ratio of the number of occurrences of each subtitle word in the subtitles in the window to the total number of subtitle words in the subtitles in the window is taken as the word frequency. Sort the word frequencies of the subtitle words in the window in descending order, and take the top 10 word frequencies as word frequency features. Extract the sentence type ratio of the subtitles in the window, where the sentence type ratio is the proportion of declarative sentences, interrogative sentences, and exclamatory sentences in the subtitles in the window. Calculate the average word count of the subtitles in the window, where the average word count is the ratio of the total number of words in the subtitles to the number of sentences in the subtitles in the window. Count the number of subtitle words in the window that are the same as preset connecting words as the window connecting words, and count the number of subtitle words in the window that are the same as preset pronouns as the window pronouns. Use the sentence type ratio, average word count, window connecting words, and window pronouns as syntactic features. A semantic vector is generated based on word frequency features and syntactic features. The expression of the semantic vector is: C(i) = 0.6 × Ftf(i) + 0.4 × Fsy(i), where C(i) represents the semantic vector, i represents the subtitle sentence number, Ftf(i) represents the word frequency feature, and Fsy(i) represents the syntactic feature. A context consistency check is performed based on the semantic vector to trigger the semantic vector regeneration mechanism; Feature extraction is performed on video data to obtain speech feature vectors and video feature vectors. Then, the semantic vector, speech feature vector, and video feature vector are fused to construct a fused feature vector. Construct a specialized terminology database and generate preliminary translation results based on the fused feature vectors; A subtitle evaluation model was constructed based on the preliminary translation results to obtain suitable subtitles; Construct a hierarchical terminology, which includes source terms, target terms, and domain tags. The source terms are the terminological expressions in the source language, the target terms are the terminological expressions in the target language corresponding to the source terms, and the domain tags are the domains to which the terms belong. The distance score is determined based on the subtitle words and the professional terminology database. The expression for the distance score is: Se(u,v)=1-ED(u,v) / max(len(u),len(v)), where Se(u,v) represents the distance score, ED(u,v) represents the edit distance between the subtitle words and the source terms in the professional terminology database, len(u) represents the character length of the subtitle words, len(v) represents the character length of the source terms, u represents the subtitle words, and v represents the source terms. When the distance score is greater than or equal to 0.85, the semantic score is determined based on the subtitle words and the professional terminology database. The synonym set of the subtitle words and the professional terminology database is extracted, and the Jaccard similarity coefficient of the synonym set is used as the semantic score. Extract the domain tags corresponding to the subtitle words in each subtitle line for domain matching judgment. If the domain tag corresponding to the subtitle word of the current subtitle line completely matches the domain tags corresponding to the subtitle words of the previous and next subtitle lines, the domain score is set to 1. If the domain tag corresponding to the subtitle word of the current subtitle line is partially related to the domain tags corresponding to the subtitle words of the previous and next subtitle lines, the domain score is set to 0.

5. If the domain tag corresponding to the subtitle word of the current subtitle line does not match the domain tags corresponding to the subtitle words of the previous and next subtitle lines, the domain score is set to 0.

2.

2. The automatic subtitle translation method based on a large model according to claim 1, characterized in that, Extract the start and end times of each subtitle from the subtitle data, and match the subtitles with their corresponding start and end times to create a timeline index table.

3. The automatic subtitle translation method based on a large model according to claim 2, characterized in that, Based on the audio signal, the speech feature vector of the audio signal corresponding to each subtitle is extracted. The frame length of the audio signal is set to 25ms and the frame shift is 10ms. The Mel frequency cepstral coefficients 2-13 of the audio signal are retained as the speech feature vector. Based on the video images, extract the video feature vector corresponding to each subtitle, extract the color histogram of the video images, take the rate of change of the proportion of different colors in the color histogram of the video images of adjacent frames as the video color feature, and count the number of faces in the video images as the video face count. The vector formed by the video color feature and the video face count is taken as the video feature vector. Feature weights are constructed based on the video scene, and a fused feature vector is constructed based on the feature weights, semantic vector, speech feature vector, and video feature vector. The semantic vector, speech feature vector, and video feature vector are reduced to the same dimension through principal component analysis, and the fused feature vector is analyzed.

4. The automatic subtitle translation method based on a large model according to claim 3, characterized in that, Matching scores are analyzed based on distance score, semantic score, and domain score. The expression for the matching score is: St(u,v)=0.4×Se(u,v)+0.3×Sj(u,v)+0.3×Sd, where St(u,v) represents the matching score, Sj(u,v) represents the semantic score, and Sd represents the domain score.

5. The automatic subtitle translation method based on a large model according to claim 4, characterized in that, The BLEU score of the initial translation results will be used as the quality score. The translated and adapted subtitles are used as historical adapted subtitles, and style features are extracted from them. The average number of words per sentence in the historical adapted subtitles, the 100 most frequent words in the historical adapted subtitles, and the sentence type ratio of the historical adapted subtitles are used as historical style features. The average number of words in the subtitles in the window, the 10 most frequent words in the subtitles in the window, and the sentence type ratio of the subtitles in the window are used as subtitle style features. The cosine similarity between the historical style features and the subtitle style features is used as style similarity. The preliminary translation results are optimized based on quality score and style similarity. When the quality score is less than 0.7, the expression of the semantic vector is updated to: C(i)=0.5×Ftf(i)+0.5×Fsy(i), and retranslation is triggered. When the style similarity is less than 0.8, the preliminary translation results are translated back to the source language according to the terminology database and retranslation is triggered. When the quality score is greater than or equal to 0.7 and the style similarity is greater than or equal to 0.8, the preliminary translation results are used as the final translation results.

6. The automatic subtitle translation method based on a large model according to claim 5, characterized in that, The translation display time is analyzed based on the final translation result. The expression for the translation display time is: Tn=To×(Lt / Ls)×(Rs / Rt), where Tn represents the translation display time, To represents the original subtitle display time, Lt represents the text length of the subtitle, Ls represents the text length of the final translation result, Rs represents the preset source language reading speed, and Rt represents the preset target language reading speed. The subtitle length revision decision is triggered based on the text length of the subtitle and the text length of the final translation result. If Lt > 1.5 × Ls, the translation length compression strategy is triggered; if Lt < 0.7 × Ls, the translation length expansion strategy is triggered. The display time of the final translation result is set to the translation display time to obtain the adapted subtitle.

7. A large-model-based automatic subtitle translation device, applied to the large-model-based automatic subtitle translation method as described in any one of claims 1-6, characterized in that, include: The data acquisition unit is used to acquire and preprocess subtitle and video data to obtain a timeline index table; The semantic analysis unit is used to build a semantic model based on the time axis index table to generate semantic vectors; The feature fusion unit is used to extract features from video data to obtain speech feature vectors and video feature vectors, and to fuse the semantic vector, speech feature vector and video feature vector to construct a fused feature vector. The subtitle translation unit is used to build a professional terminology database and generate preliminary translation results based on the fused feature vectors. The translation evaluation unit is used to build a subtitle evaluation model based on the preliminary translation results in order to obtain suitable subtitles.

8. A storage medium, characterized in that, The system stores instructions that, when executed on a computer, cause the computer to perform the automatic subtitle translation method based on a large model as described in any one of claims 1-6.

Images