Subtitle annotation method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of computers, in particular to a subtitle pinyin annotation method and device, electronic equipment and a storage medium, which can be applied to cloud technology, artificial intelligence, intelligent transportation, auxiliary driving and the like, so as to improve the pinyin annotation accuracy of Chinese subtitles. The method comprises the following steps: obtaining a plurality of candidate pinyins respectively associated with at least one polyphonic character contained in a subtitle text; extracting a text semantic feature of the subtitle text, and obtaining a semantic feature corresponding to the at least one polyphonic character based on the text semantic feature; extracting an audio feature of an audio segment corresponding to the subtitle text; and performing the following operations on the at least one polyphonic character respectively: determining a matching probability corresponding to each of the plurality of candidate pinyins associated with one polyphonic character based on the semantic feature and the audio feature of the one polyphonic character, and selecting a target pinyin from the plurality of candidate pinyins based on the plurality of matching probabilities to annotate the one polyphonic character; thus, the Chinese subtitles can be accurately annotated, and the pinyin annotation accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a subtitle phonetic annotation method, apparatus, electronic device, and storage medium. Background Art

[0002] With the rapid development of video applications, various videos emerge in an endless stream, enriching people's lives. Since young children have limited literacy and need to rely on pinyin to understand the content when watching videos, some videos targeted at young children usually use pinyin to annotate Chinese subtitles.

[0003] In the related art, when using pinyin to annotate Chinese subtitles of videos, the text of the Chinese subtitles is usually input into a deep learning model, and the pinyin of each character in the text is predicted separately by the deep learning model, and a pinyin sequence corresponding to the Chinese subtitles is output.

[0004] However, due to the existence of polyphonic characters in Chinese subtitles, the above deep learning model is prone to mislabel the pinyin of polyphonic characters, resulting in inaccurate pinyin annotation for Chinese subtitles; for example Figure 1 as shown, the Chinese subtitle is "Now it's time to repair the Octopod together", and the corresponding pinyin sequence is "Xian Zai Gai Yi Qi Xiu Li Zhang Yu Pu Le". Among them, the pinyin of the character "堡" is mislabeled as "Pu", and the correct pinyin should be "bao".

[0005] Incorrect phonetic annotation will affect young children's understanding of video content. Seriously, it may mislead young children's literacy cognition and have a negative impact on the video application platform. Therefore, how to ensure the accuracy of phonetic annotation for Chinese subtitles is a problem that needs to be solved. Summary of the Invention

[0006] Embodiments of this application provide a subtitle phonetic annotation method, apparatus, electronic device, and storage medium for improving the accuracy of phonetic annotation for Chinese subtitles.

[0007] On the one hand, embodiments of this application provide a subtitle phonetic annotation method, including:

[0008] Obtaining multiple candidate pinyins associated with each of at least one polyphonic character included in the subtitle text;

[0009] Performing semantic feature extraction on the subtitle text to obtain a text semantic feature corresponding to the subtitle text, and obtaining a semantic feature corresponding to each of the at least one polyphonic character based on the text semantic feature;

[0010] Performing audio feature extraction on an audio segment corresponding to the subtitle text to obtain an audio feature corresponding to the audio segment;

[0011] For each of the at least one polyphonic character, the following operations are performed: based on the semantic features and audio features of a polyphonic character, determine the matching probability of each of the multiple candidate pinyin associated with the polyphonic character, and select the target pinyin from the multiple candidate pinyin based on the multiple matching probabilities to annotate the polyphonic character.

[0012] On one hand, embodiments of this application provide a subtitle phonetic annotation device, including:

[0013] The acquisition module is used to acquire multiple candidate pinyin associated with each of the at least one polyphonic character contained in the subtitle text;

[0014] The text feature extraction module is used to extract semantic features from the subtitle text, obtain the text semantic features corresponding to the subtitle text, and obtain the semantic features corresponding to each of the at least one polyphonic character based on the text semantic features;

[0015] The audio feature extraction module is used to extract audio features from the audio segment corresponding to the subtitle text to obtain the audio features corresponding to the audio segment.

[0016] The polyphonic character annotation module is used to perform the following operations for the at least one polyphonic character: based on the semantic features and audio features of a polyphonic character, determine the matching probability of each of the multiple candidate pinyin associated with the polyphonic character, and select the target pinyin from the multiple candidate pinyin based on the multiple matching probabilities to annotate the polyphonic character.

[0017] Optionally, when determining the matching probability of each of the multiple candidate pinyin associated with a polyphonic character based on its semantic features and audio features, the polyphonic character phonetic annotation module is further configured to:

[0018] An attention mechanism is used to process the semantic features and audio features of the polyphonic character to obtain interactive features;

[0019] Based on the semantic features, audio features, and interaction features of the polyphonic character, the matching probability of each of the multiple candidate pinyin associated with the polyphonic character is determined.

[0020] Optionally, the device further includes:

[0021] The speech recognition module is used to perform speech recognition on the audio segment corresponding to the subtitle text, obtain the pronunciation sequence corresponding to the audio segment and the recognition probability of each pronunciation, and obtain the target pronunciation corresponding to the polyphonic character from the pronunciation sequence;

[0022] When selecting the target pinyin from the multiple candidate pinyin based on multiple matching probabilities and annotating the polyphonic character, the polyphonic character annotation module is further used for:

[0023] If there is a candidate pinyin among the multiple candidate pinyin that matches the target pronunciation, then the matching probability of the matching candidate pinyin is updated based on the recognition probability of the target pronunciation to obtain the updated matching probability;

[0024] From the multiple candidate pinyin, the candidate pinyin whose matching probability satisfies the probability condition is selected as the target pinyin, and the polyphonic character is marked.

[0025] Optionally, the device further includes a matching module, configured to determine, from a plurality of candidate pinyin, the candidate pinyin that matches the target pronunciation in the following manner:

[0026] For each of the multiple candidate pinyin, perform the following operations:

[0027] Select a phrase containing the polyphonic character from the subtitle text, and determine the pinyin sequence corresponding to the phrase, wherein the pinyin sequence contains a candidate pinyin corresponding to the polyphonic character;

[0028] If the pinyin sequence matches the pronunciation sequence corresponding to the audio segment, then the candidate pinyin is used as a candidate pinyin that matches the target pronunciation.

[0029] Optionally, when updating the matching probability of the candidate pinyin based on the recognition probability of the target pronunciation to obtain the updated matching probability, the polyphonic character phonetic notation module is further used to:

[0030] From the pronunciation sequences corresponding to the audio segment, select the first pronunciation sequence containing the target pronunciation; wherein, the first pronunciation sequence is the same as the second pronunciation sequence after the pinyin sequence is converted;

[0031] Based on the recognition probabilities of multiple pronunciations in the first pronunciation sequence, the matching probabilities of the candidate pinyin are updated to obtain the updated matching probabilities.

[0032] Optionally, the text feature extraction module is further used for:

[0033] The subtitle text is input into a pre-trained language model to obtain the semantic features of the text;

[0034] For each of the at least one polyphonic characters, perform the following operations: based on the position information of the polyphonic character in the subtitle text, select the semantic feature corresponding to the position information from the text semantic features.

[0035] Optionally, the audio feature extraction module is further configured to:

[0036] The audio segment corresponding to the subtitle text is subjected to Fourier transform processing to obtain the spectrum diagram corresponding to the audio segment;

[0037] The multiple audio frames obtained based on the spectrogram are input into the audio feature extraction model to obtain the audio frame features corresponding to each of the multiple audio frames.

[0038] The obtained features of multiple audio frames are used as the audio features corresponding to the audio segment.

[0039] Optionally, the audio feature extraction model includes a deep representation module and an attention-based feature extraction module;

[0040] When inputting multiple audio frames obtained based on the spectrogram into the audio feature extraction model to obtain the audio frame features corresponding to each of the multiple audio frames, the audio feature extraction module is further used for:

[0041] The multiple audio frames obtained based on the spectrogram are input into the depth representation module to obtain the intermediate audio frame features corresponding to each of the multiple audio frames.

[0042] The obtained intermediate audio frame features are input into the attention-based feature extraction module to obtain the audio frame features corresponding to each of the multiple audio frames.

[0043] Optionally, the subtitle text further includes at least one monosyllabic character, and the device further includes a monosyllabic character annotation module for:

[0044] Obtain a pinyin associated with each of the at least one monosyllabic character;

[0045] For each of the at least one monosyllabic characters, perform the following operations: use a pinyin associated with a monosyllabic character to annotate the monosyllabic character.

[0046] On one hand, embodiments of this application provide an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described subtitle phonetic annotation methods.

[0047] On one hand, embodiments of this application provide a computer storage medium including a computer program, which, when run on an electronic device, causes the electronic device to perform the steps of any of the above-described subtitle phonetic annotation methods.

[0048] On one hand, embodiments of this application provide a computer program product, which includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described subtitle phonetic annotation methods.

[0049] Since the embodiments of this application adopt the above-described technical solution, they have at least the following technical effects:

[0050] In the solution of this application embodiment, for subtitle text containing polyphonic characters, when annotating polyphonic characters, firstly, multiple candidate pinyin associated with the polyphonic characters are obtained; then, by extracting semantic features from the subtitle text, the semantic features corresponding to the polyphonic characters can be obtained; simultaneously, the audio features of the audio segment corresponding to the subtitle text are extracted; based on the semantic features of the polyphonic characters and the aforementioned audio features, the matching probability corresponding to each of the multiple candidate pinyin associated with the polyphonic characters is determined, and then the target pinyin is selected from the multiple candidate pinyin based on the multiple matching probabilities to annotate a polyphonic character. In this way, by combining the semantic features and audio features of polyphonic characters, the matching probability of multiple candidate pinyin of polyphonic characters can be accurately predicted, thereby selecting the target pinyin of the polyphonic character based on the matching probability, making the target pinyin more consistent with the context of the subtitle text, so as to accurately annotate the polyphonic characters, thereby improving the accuracy of phonetic annotation in subtitle text, more accurately assisting users to better understand video content, and improving the product experience of video platforms.

[0051] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram illustrating the phonetic annotation of a video subtitle in a related technology;

[0054] Figure 2 This is a schematic diagram illustrating an application scenario of a subtitle phonetic annotation method provided in an embodiment of this application;

[0055] Figure 3A flowchart of a subtitle phonetic annotation method provided in this application embodiment;

[0056] Figure 4 A flowchart illustrating another method for phonetic subtitle annotation provided in this application embodiment;

[0057] Figure 5 This application provides a schematic diagram of semantic feature extraction for subtitle text.

[0058] Figure 6 A flowchart illustrating another method for phonetic subtitle annotation provided in this application embodiment;

[0059] Figure 7 This is a schematic diagram of audio feature extraction for an audio segment provided in an embodiment of this application;

[0060] Figure 8 A schematic diagram of the structure of a polyphonic character phonetic notation model provided in an embodiment of this application;

[0061] Figure 9 This application provides a schematic diagram of speech recognition for an audio segment.

[0062] Figure 10 A flowchart illustrating another method for phonetic subtitle annotation provided in this application embodiment;

[0063] Figure 11 A logical diagram illustrating a phonetic subtitle method provided in an embodiment of this application;

[0064] Figure 12A A schematic diagram of phonetic annotation for video subtitles provided in this application embodiment;

[0065] Figure 12B This is an illustration of another phonetic transcription for video subtitles provided in an embodiment of this application;

[0066] Figure 13 A structural block diagram of a subtitle phonetic annotation device provided in this application embodiment;

[0067] Figure 14 A structural block diagram of a subtitle phonetic annotation device provided in this application embodiment;

[0068] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0069] Figure 16 This is a schematic diagram of the structure of another electronic device in an embodiment of this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following will further describe this application in detail with reference to the accompanying drawings. Apparently, the described embodiments are only a part of the embodiments of this application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0071] To facilitate a better understanding of the technical solutions of this application by those skilled in the art, the following introduces the terms related to this application.

[0072] 1) Subtitle: It refers to the non-video content such as dialogue or narration displayed in the form of text in online videos, TVs, movies, and stage works, and generally refers to the text processed in the later stage of film and television works.

[0073] 2) Subtitle phonetic annotation: Use pinyin to mark the Chinese characters in the subtitle.

[0074] 3) Speech recognition technology: Also known as Automatic Speech Recognition (ASR), its goal is to recognize the lexical content in human speech into text information, and the text information can be used as computer-readable input, such as keystrokes, binary codes, or character sequences. Speech recognition technology is widely used in scenarios such as voice dialing, voice navigation, smart home control, voice search, dictation data entry, etc. Each word or phrase corresponds to a pronunciation, for example: the pronunciation corresponding to "你好" is "ni hao".

[0075] 4) Attention mechanism: It originates from the research on human vision. In cognitive science, due to the bottleneck of information processing, humans will selectively focus on a part of all information while ignoring other visible information. The above mechanism is usually called the attention mechanism. The attention mechanism can enable the neural network to have the ability to focus on a subset of its input (or features): select specific input. In the case of limited computing power, the attention mechanism is a resource allocation scheme for solving the problem of information overload, allocating computing resources to more important tasks.

[0076] The term "exemplary" used hereinafter means "serving as an example, embodiment, or illustration". Any embodiment described as "exemplary" does not have to be construed as superior or better than other embodiments.

[0077] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0078] This application relates to the field of artificial intelligence (AI) technology, and is designed based on natural language processing (NLP), speech processing and machine learning technologies in artificial intelligence.

[0079] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0080] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0081] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Compared to data mining, which searches for patterns in large datasets, machine learning focuses more on algorithm design, enabling computers to automatically "learn" patterns from data and use these patterns to predict unknown data. Machine learning is the core of artificial intelligence and the fundamental way to give computers intelligence; deep learning is the core of machine learning and a technology for implementing it. Machine learning typically includes techniques such as deep learning, reinforcement learning, transfer learning, and inductive learning. Deep learning includes techniques such as MobileNet (mobile vision neural networks), Convolutional Neural Networks (CNNs), Deep Belief Networks, Recurrent Neural Networks, Autoencoders, and Generative Adversarial Networks.

[0082] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0083] Key technologies in speech processing technology include automatic speech recognition, speech synthesis, and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech being one of the most promising methods.

[0084] The pre-trained language model in this application embodiment is built based on machine learning and natural language processing technology, and the audio feature extraction model is built based on machine learning and speech processing technology.

[0085] In related technologies, when using pinyin to annotate Chinese subtitles in videos, the text of the Chinese subtitles is usually input into a deep learning model. The deep learning model then predicts the pinyin for each character in the text and outputs the corresponding pinyin sequence for the Chinese subtitles. However, because Chinese subtitles contain polyphonic characters, the aforementioned deep learning model is prone to misannotating the pinyin of polyphonic characters, resulting in inaccurate pinyin annotations for the Chinese subtitles.

[0086] Considering that the audio corresponding to the subtitle text can help with phonetic annotation of the subtitle text, combining the subtitle text with the corresponding audio can make the phonetic annotation of the subtitle text more consistent with the pronunciation context of the video, which is better than using the subtitle text alone for phonetic annotation.

[0087] In view of this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for annotating subtitles with phonetic symbols. By combining the semantic features and audio features of polyphonic characters, the matching probability of multiple candidate pinyin for polyphonic characters can be accurately predicted, thereby selecting the target pinyin for polyphonic characters based on the matching probability, so as to accurately annotate polyphonic characters.

[0088] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0089] like Figure 2 The diagram shown illustrates an application scenario in this embodiment of the application. This application scenario diagram includes multiple terminal devices 210 and a server 220. The terminal devices 210 and the server 220 can communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network. The terminal devices 210 and the server 220 can be directly or indirectly connected via wired or wireless communication methods; this application does not impose any restrictions on this.

[0090] In this application embodiment, the terminal device 210 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, in-vehicle terminals, and other devices. The terminal device may have a video-related client installed, which can be software (such as video playback software, live video streaming software, etc.), or a webpage, mini-program, etc. The server 220 can be a backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for annotating video subtitles; this application does not impose specific limitations. The server 220 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0091] It should be noted that the methods in the various embodiments of this application can be executed by an electronic device, which can be a server 220 or a terminal device 210. That is, the method can be executed by the server 220 or the terminal device 210 alone, or by both the server 220 and the terminal device 210 together. When the server 220 or the terminal device 210 executes the method alone, for example, the server 220 or the terminal device 210 can obtain the subtitle text in the video, then determine the pinyin of each character in the subtitle text, and use the obtained pinyin sequence to annotate the subtitle text. When the server 220 and the terminal device 210 execute the method together, for example, the server 220 can obtain the subtitle text of the video, then determine the pinyin of each character in the subtitle text, and send the video and the pinyin sequence corresponding to the subtitle text to the terminal device, which then uses the pinyin sequence to annotate the subtitle text when playing the video. The following example illustrates the method executed by the server alone.

[0092] In some embodiments, the subtitle annotation method of this application can be applied to video playback scenarios. For example, a video playback software is installed on a terminal device 210, and a server 220 is the backend server corresponding to the video playback software. The server 220 stores various recorded videos, including but not limited to television videos and movie videos. These videos can carry subtitles, and the subtitle annotation method of this application can be used to annotate the subtitles of these videos. When a user plays a video on the terminal device 210 using the video playback software, the terminal device 210 can send a video acquisition request to the server 220. Then, the server 220 can send the video corresponding to the video acquisition request to the terminal device 210. The video contains subtitles and the corresponding annotations.

[0093] This application embodiment does not limit the number of servers 220 and terminal devices 210. Those skilled in the art will understand that the number of servers 220 and terminal devices 210 can be flexibly selected in different application scenarios. For example, when the data processing volume is large and the complexity is high, multiple servers 220 can be used for processing; when multiple users are watching video content, the method of this application embodiment can be applied to multiple terminal devices 210.

[0094] It should be noted that, Figure 2 This paper provides illustrative examples of the application scenarios for the subtitle annotation method of this application. However, the application scenarios applicable to the method in the actual embodiments of this application are not limited to these. Furthermore, the embodiments of this application can be applied to various scenarios, including not only video playback scenarios, but also scenarios involving cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0095] The specific implementation method of the subtitle phonetic annotation method of this application will be described below.

[0096] Figure 3 The figure shows a schematic diagram of a subtitle phonetic annotation method provided by an embodiment of the present application. This method can be executed by a terminal device or a server, or jointly executed by a server and a terminal device. Hereinafter, taking the execution by the server as an example for illustration. As Figure 3 shown, the subtitle phonetic annotation method of the embodiment of the present application may include the following steps S301 - S304:

[0097] Step S301: Obtain multiple candidate pinyins respectively associated with at least one polyphonic character included in the subtitle text.

[0098] Among them, for a video with subtitles, a sentence of subtitles in the same picture can be used as a subtitle text. For each obtained subtitle text, word segmentation can be performed on the subtitle text to obtain multiple characters, and then the pinyin corresponding to each character can be determined to phonetically annotate each character with pinyin.

[0099] Since the set of pinyins corresponding to each character in Chinese is fixed, this set of pinyins may contain one pinyin or multiple pinyins. Therefore, the association relationship between characters and pinyins can be preset; among them, the association relationship between characters and pinyins can be saved in the form of a table. For example, as shown in Table 1, if a character is associated with one pinyin, then this character is a monosyllabic character; if a character is associated with multiple pinyins, then this character is a polyphonic character.

[0100] It should be noted that one or more pinyins associated with each character can be with tones or without tones; when with tones, when one pinyin associated with a character contains multiple tones, this character is also a polyphonic character. For example, the pinyin corresponding to the character "挨" is "ai", but it contains two tones, the first tone and the second tone. In the above Table 1, the example is illustrated with pinyins without tones.

[0101] Table 1

[0102] Can ke Monosyllabic characters lock up quan / juan Polyphonic characters Qian gan / qian Polyphonic characters Fort bao / pu / bu Polyphonic characters quack gua / gu Polyphonic characters ... ... ...

[0103] In step S301, for a subtitle text containing at least one polyphonic character, the multiple pinyins associated with each polyphonic character can be determined by querying the above-mentioned association relationship between characters and pinyins, and these multiple pinyins are used as multiple candidate pinyins. Then, the target pinyin is selected from the multiple candidate pinyins of each polyphonic character through the following steps S302 - S304.

[0104] In some embodiments, a subtitle text may not only contain polyphonic characters but also at least one monosyllabic character. When phonetically annotating this subtitle text, the one pinyin associated with each monosyllabic character can be obtained through the above-mentioned association relationship between characters and pinyins, and then the obtained pinyin is used to phonetically annotate the monosyllabic character.

[0105] Therefore, before performing the above step S301, each single Chinese character in the subtitle text can also be phonetically annotated, and the following steps A1 - A2 can be specifically performed:

[0106] Step A1: Obtain one pinyin associated with each of at least one single Chinese character.

[0107] Step A2: Perform the following operations for each of the at least one single Chinese character respectively: Use one pinyin associated with a single Chinese character to annotate the single Chinese character.

[0108] For example, the subtitle text is "Look, the Octo-Star is repaired", which contains multiple single Chinese characters, namely: 看 (Kan), 章 (Zhang), 鱼 (Yu), 修 (Xiu), 理 (Li), 好 (Hao), and contains three polyphonic characters, namely: 堡 (Bao / Pu / Bu), 了 (Le / Liao); it should be noted that here the pinyin of each character is without tones. If the pinyin of each character has tones, then the character "好" is also a polyphonic character.

[0109] For each of the above single Chinese characters, phonetically annotate it with the corresponding pinyin of the single Chinese character; for each of the above polyphonic characters, obtain multiple candidate pinyins corresponding to the polyphonic character. Further, phonetically annotate each polyphonic character in the subtitle text through the following steps S302 - S304.

[0110] Step S302: Extract semantic features of the subtitle text, obtain the text semantic features corresponding to the subtitle text, and obtain the semantic features corresponding to each of at least one polyphonic character based on the text semantic features.

[0111] In this step, a language model can be used to extract semantic features of the subtitle text. For example, the language model can be a pre-trained language model or other language models, and the embodiments of the present application do not limit this. The following takes the pre-trained language model as an example for illustration.

[0112] In some possible implementation manners, as Figure 4 shown, the above step S302 can specifically include the following steps S3021 - S3022:

[0113] Step S3021: Input the subtitle text into the pre-trained language model to obtain text semantic features.

[0114] Step S3022: Perform the following operations for each of at least one polyphonic character respectively: Based on the position information of a polyphonic character in the subtitle text, select the semantic features corresponding to the position information from the text semantic features.

[0115] Among them, the pre-trained language model can adopt the BERT (Bidirectional Encoder Representations) model, or other types of pre-trained language models, which are not limited here. Below, the pre-trained language model is taken as the BERT model for illustration.

[0116] As Figure 5 shown, still taking the subtitle text "Look, the Octopod is repaired" as an example, before inputting the subtitle text into the BERT model, first, according to the word position representation rule in the BERT model, determine the word position representation of each character in the subtitle text, and the word position representation of each character is used to explain the word where each character is located; then, according to the character position representation rule in the BERT model, determine the position representation of each character in the subtitle text, and the position representation of each character is used to explain the position of each character in the subtitle text; then, according to the correspondence between the character and the vector in the BERT model, convert each character into the corresponding character representation; finally, sum up the corresponding word position representation, character position representation and character representation of each character to obtain the input feature of the BERT model.

[0117] After inputting the above input features into the BERT model, the text representation of the subtitle text (including the representation of each character) is output, and this text representation can be used as the text semantic feature; based on the position information of each polyphonic character in the subtitle text, the representation of each polyphonic character (i.e., the semantic feature) can be obtained. For example Figure 5 the representation of the polyphonic character "bao" and the representation of the polyphonic character "le" output in

[0118] In this step, the subtitle text is modeled by the pre-trained language model, so that the semantic features of the output polyphonic characters can fully capture the context of the subtitle text, making the target pinyin of the subsequent output polyphonic characters more consistent with the context in the subtitle text.

[0119] Step S303: Extract audio features from the audio segment corresponding to the subtitle text to obtain the audio features corresponding to the audio segment.

[0120] Among them, the audio segment corresponding to the subtitle text can be preprocessed to obtain the spectrogram corresponding to the audio segment, and multiple audio frames are obtained from the spectrogram; then an audio feature extraction model is used to extract audio features from each of the multiple audio frames respectively to obtain the audio frame features of each of the multiple audio frames, and then the audio features corresponding to the audio segment can be accurately obtained.

[0121] In some possible implementation manners, as Figure 6 shown, the above step S303 may specifically include the following steps S3031 - S3033:

[0122] Step S3031: Perform Fourier transform on the audio segment corresponding to the subtitle text to obtain the spectrum of the audio segment.

[0123] The audio data in the audio segment is in the time domain, that is, the horizontal axis is time and the vertical axis is the intensity of the audio signal. By performing Fourier transform on the audio segment, the audio signal can be converted from the time domain to the frequency domain, that is, the audio signal is decomposed into sine waves of different frequencies to obtain a spectrum, which can be a Mel spectrum.

[0124] Step S3032: Input the multiple audio frames obtained based on the spectrogram into the audio feature extraction model to obtain the audio frame features corresponding to each of the multiple audio frames.

[0125] In the spectrogram, a set time window can be considered as an audio frame, thus obtaining multiple audio frames. The set time window can be set as needed, for example, 0.96s, without limitation. The audio feature extraction model can first obtain the depth representation of each of the multiple audio frames, and then extract the audio frame features of each audio frame based on the depth representation of each audio frame.

[0126] Step S3033: The obtained multiple audio frame features are used as the audio features corresponding to the audio segments.

[0127] The structure of the above audio feature extraction model will be described below as an example.

[0128] In some embodiments, the audio feature extraction model described above may include a depth representation module and an attention-based feature extraction module; the depth representation module is used to obtain the depth representation of each of the multiple audio frames, and the attention-based feature extraction module is used to extract the audio frame features of each audio frame.

[0129] Based on this, step S3032 above may include the following steps B1-B2:

[0130] Step B1: Input the multiple audio frames obtained based on the spectrogram into the depth representation module to obtain the intermediate audio frame features corresponding to each of the multiple audio frames.

[0131] The module inputs multiple audio frames into the depth representation module and outputs the depth representation of each audio frame. The depth representation of each audio frame is used as the intermediate audio frame feature of that audio frame.

[0132] For example, the deep representation module can employ a deep learning model, such as the VGGish model, without limitation. The VGGish model is a VGG (Visual Geometry Group, super-resolution test sequence) model pre-trained on a specified dataset. The VGG model is a convolutional neural network model. Specifically, the VGGish model can transform input audio features into semantically meaningful 128-dimensional feature vectors.

[0133] Step B2: Input the obtained intermediate audio frame features into the attention-based feature extraction module to obtain the audio frame features corresponding to each audio frame.

[0134] In this step, the feature extraction module based on the attention mechanism can perform temporal interaction on the features of multiple audio frames, so that each output audio frame feature can capture the contextual information of the audio segment.

[0135] For example, an attention-based feature extraction module can employ a Transformer architecture. The Transformer is a feature extractor implemented using an attention mechanism, which can be used in an encoder-decoder architecture, or applied independently to the encoder or decoder. The following embodiments in this application use a Transformer-Encoder architecture as an example.

[0136] For example, such as Figure 7 As shown, multiple audio frames of the audio segment corresponding to the subtitle text are input into the VGGish model to obtain the depth representation (i.e., intermediate audio frame features) of each audio frame. Then, the depth representation of each audio frame is input into the Transformer-Encoder model, which performs temporal interaction on the features of multiple audio frames and outputs the representation of each audio frame (i.e., audio frame features).

[0137] Step S304: Perform the following operations for at least one polyphonic character: Based on the semantic features and audio features of a polyphonic character, determine the matching probability of each of the multiple candidate pinyin associated with the polyphonic character, and select the target pinyin from the multiple candidate pinyin based on the multiple matching probabilities to annotate the polyphonic character.

[0138] Specifically, based on the semantic features of a polyphonic character and the audio features of the audio segment corresponding to the subtitle text, a prediction network model (such as a classification model) can be used to predict the matching probability of each of the multiple candidate pinyin associated with the polyphonic character; then, from the multiple candidate pinyin, the candidate pinyin with the highest matching probability is selected as the target pinyin, and the polyphonic character is labeled; the target pinyin is the pinyin that conforms to the context of the subtitle text.

[0139] In this embodiment, when annotating the subtitle text in a video with pinyin, the subtitle text is combined with the corresponding audio segment. A polyphonic character pinyin model is used to comprehensively model the subtitle text and audio segment, thereby more accurately predicting the target pinyin of polyphonic characters in the subtitle text. This makes the target pinyin more consistent with the context of the subtitle text, thus improving the accuracy of the pinyin annotation. This can more accurately help users better understand the video content and improve the product experience of the video platform.

[0140] In some embodiments, in order to further accurately predict the matching probability of multiple candidate pinyin associated with each polyphonic character, the above step S304, based on the semantic features and audio features of a polyphonic character, determines the matching probability of each of the multiple candidate pinyin associated with a polyphonic character, which may include the following steps C1-C2:

[0141] Step C1: Use an attention mechanism to process the semantic and audio features of a polyphonic character to obtain interactive features.

[0142] The audio features include the audio frame features of multiple audio frames. Attention is calculated on the semantic features of the polyphonic characters and the multiple audio frame features. The calculation process includes: calculating the similarity between the multiple audio frame features and the semantic features of the polyphonic characters, performing probability normalization on the obtained similarities so that the sum of the normalized similarities is 1, that is, obtaining the similarity distribution between the multiple audio frame features and the semantic features of the polyphonic characters, multiplying the similarity distribution by the multiple audio frame features respectively and summing the results to obtain the above-mentioned interaction features.

[0143] For example, the similarity distribution between features of multiple audio frames and semantic features of polyphonic characters includes S1, S2...S... n S1+S2+......+S n =1, multiple audio frame features including a1, a2...a n Then the interaction feature Y = S1*a1 + S2*a2 + ... + S n *a n .

[0144] Step C2: Based on the semantic features, audio features, and interaction features of a polyphonic character, determine the matching probability of each of the multiple candidate pinyin associated with the polyphonic character.

[0145] Here, the semantic features, audio features, and interaction features of a polyphonic character are concatenated and input into a prediction network model, which can accurately predict the matching probability of each of the multiple candidate pinyin associated with the polyphonic character.

[0146] In this embodiment, the pre-trained language model and audio feature extraction model described above can constitute a polyphonic character phonetic annotation model. This polyphonic character phonetic annotation model can be trained based on a training sample set. Each training sample in the training sample set includes: sample subtitle text annotated with pinyin and the audio segment corresponding to the sample subtitle text. The following describes the process in conjunction with... Figure 8 The structure of the phonetic model for polyphonic characters is described in the example, which illustrates the prediction process of multiple candidate pinyin for polyphonic characters.

[0147] like Figure 8 As shown, the left side of the polyphonic character phonetic model is a pre-trained language model, such as the BERT model. The BERT model is used to model the subtitle text of the video to obtain the text representation of the subtitle text (i.e., text semantic features). From the text representation, the polyphonic character representation (i.e., the semantic features of the polyphonic character) can be obtained. In this way, the context of the polyphonic characters in the subtitle text can be fully captured, so that the target pinyin of the subsequently output polyphonic characters is more consistent with the context of the subtitle text.

[0148] The right side of the polyphonic character phonetic annotation model is the audio feature extraction model, which includes models such as the VGGish model and the Transformer-Encoder model. This model can model the video segments corresponding to the subtitle text. First, the video segment is subjected to a Fourier transform, and then a Mel-spectrum is constructed. An audio frame is defined as a time window (e.g., 0.96s). Each audio frame is used to construct a deep representation of the audio frame through the VGGish model. Then, the Transformer-Encoder model uses its self-attention mechanism to perform temporal interaction on multiple audio frames, obtaining multiple audio frame representations (i.e., audio frame features). In this way, each output audio frame representation can capture the contextual information of the audio segment.

[0149] Further, after obtaining the representations of polyphonic characters in the subtitle text and the representations of multiple audio frames of the video clip, attention calculation can be performed on the representations of polyphonic characters in the subtitle text and the representations of multiple audio frames to obtain interaction features; finally, the representations of polyphonic characters, interaction features, and representations of multiple audio frames (Maxpooling) are concatenated and input into the prediction network model to predict the matching probabilities P_Y of multiple candidate pinyins associated with the polyphonic characters in the subtitle text. For example, the matching probabilities of each candidate pinyin of the character "堡" are: P_Y("Bao"|"堡") = 0.829, P_Y("Pu"|"堡") = 0.163, P_Y("Bu"|"堡") = 0.008.

[0150] Based on the above embodiments of the present application, in order to further improve the accuracy of polyphonic character pronunciation, speech recognition can be performed on the audio clip corresponding to the subtitle text to obtain the pronunciation sequence corresponding to the audio clip, and then, through this pronunciation sequence, the matching probabilities of the corresponding candidate pinyins of each polyphonic character in the above step S304 are updated, and the corresponding candidate pinyin matches the pronunciation sequence.

[0151] Therefore, before selecting a candidate pinyin as the target pinyin based on multiple matching probabilities in the above step S304 to annotate a polyphonic character, the following steps can also be performed:

[0152] Perform speech recognition on the audio clip corresponding to the subtitle text to obtain the pronunciation sequence corresponding to the audio clip and the recognition probability of each pronunciation, and obtain the target pronunciation corresponding to a polyphonic character from the pronunciation sequence.

[0153] Among them, speech recognition technology ASR can be used to perform speech recognition on the video clip corresponding to the subtitle text. Specifically, as Figure 9 shown, the video clip corresponding to the subtitle text is input into the ASR model. If a pronunciation sequence is output, it can further assist in pinyin annotation of polyphonic characters in the subtitle text. For example: for the video clip corresponding to the subtitle text "Look, the Octopus Fort is repaired", after speech recognition by the ASR model, the output pronunciation sequence is: Kan|0.81, Zhang|0.76, Yu|0.65, Bao|0.79, Xiu|0.88, Li|0.96, Hao|0.86, Le|0.80; where the value after each pronunciation represents the recognition probability of that pronunciation, that is, the recognition probability of Kan is 0.81......the recognition probability of Le is 0.80.

[0154] Further, in the above step S304, selecting a target pinyin from multiple candidate pinyins based on multiple matching probabilities to annotate a polyphonic character may include the following steps D1 - D2:

[0155] Step D1: If there is a candidate pinyin that matches the target pronunciation among multiple candidate pinyins, update the matching probability of the matching candidate pinyin based on the recognition probability of the target pronunciation to obtain the updated matching probability.

[0156] In this step, multiple candidate pinyins of a polyphonic character can be compared with the target pronunciation corresponding to this polyphonic character respectively to determine whether there is a candidate pinyin that matches the target pronunciation. For example: For the polyphonic character "堡" in the subtitle text, its multiple candidate pinyins include "Bao", "Pu", "Bu". Suppose the target pronunciation corresponding to this polyphonic character in the pronunciation sequence corresponding to the audio segment is "Bao", then the candidate pinyin "Bao" matches the target pronunciation.

[0157] After obtaining the candidate pinyin that matches the target pronunciation, update the matching probability of this matching candidate pinyin based on the recognition probability of the target pronunciation. Exemplarily, the matching probability of the candidate pinyin "Bao" is P_Y("bao"|"堡") = 0.829, and the recognition probability of the target pronunciation "bao" is ASR(Bao|0.79), then the updated matching probability P_YF("Bao"|"堡") = min(1.0, 0.829 + ASR(Bao|0.79)) = 1.0. In this way, the target pinyin of the polyphonic character "堡" in the subtitle text is "Bao".

[0158] Step D2: Select a candidate pinyin whose matching probability meets the probability condition from multiple candidate pinyins as the target pinyin to annotate a polyphonic character.

[0159] Among them, the candidate pinyin with the maximum matching probability can be used as the target pinyin to annotate the corresponding polyphonic character. For example: For the polyphonic character "堡", the matching probabilities of its multiple candidate pinyins obtained through the polyphonic character phonetic annotation model are: P_Y("Bao"|"堡") = 0.829, P_Y("Pu"|"堡") = 0.163, P_Y("Bu"|"堡") = 0.008; Further, after updating P_Y("Bao"|"堡") through the above step D2, the updated matching probability P_YF("bao"|"堡") = 1.0 is obtained. Therefore, the matching probability of "Bao" is the largest, that is, the target pinyin corresponding to "堡" is "Bao".

[0160] Optionally, when determining whether there is a candidate pinyin that matches the target pronunciation among multiple candidate pinyins in the above step D1, in order to further improve the matching accuracy between multiple candidate pinyins and the target pronunciation, for each candidate pinyin of the polyphonic character, determine the pinyin sequence of the phrase where the polyphonic character is located, and match this pinyin sequence with the pronunciation sequence where the target pronunciation is located.

[0161] At this time, when determining whether there is a candidate pinyin that matches the target pronunciation among multiple candidate pinyins, the following steps E1 - E2 can be respectively executed for the multiple candidate pinyins:

[0162] Step E1: Select a phrase containing a polyphonic character from the subtitle text, and determine the corresponding pinyin sequence. The pinyin sequence contains a candidate pinyin corresponding to the polyphonic character.

[0163] Exemplarily, through the comparison method of Trigram, that is, taking the polyphonic character and one character from the context of the polyphonic character as a phrase containing the polyphonic character. For example, in the subtitle text "Look, the Octo Star is repaired", the phrase containing "Star" is "Octo Star"; the pinyin sequence where a candidate pinyin "Bao" is located is "Zhang, Yu, Bao".

[0164] Step E2: If the pinyin sequence matches the pronunciation sequence corresponding to the audio clip, then take a candidate pinyin as the candidate pinyin that matches the target pronunciation.

[0165] In this step, if the pinyin sequence corresponding to the polyphonic word phrase can be determined to match the pronunciation sequence corresponding to the audio clip in the pronunciation sequence corresponding to the audio clip.

[0166] For example: For the polyphonic word phrase "Octo Star" in the above subtitle text "Look, the Octo Star is repaired", its corresponding pinyin sequence is "Zhang, Yu, Bao", and the pronunciation sequence is "Kan, Zhang, Yu, Bao, Xiu, Li, Hao, Le". It can be seen that the pinyin sequence "Zhang, Yu, Bao" is in the pronunciation sequence. Therefore, the candidate pinyin "Bao" matches the target pronunciation "Bao" in the pronunciation sequence.

[0167] Furthermore, in the above step D1, based on the recognition probability of the target pronunciation, updating the matching probability of the matching candidate pinyin to obtain the updated matching probability may include the following steps D11 - D12:

[0168] Step D11: Select the first pronunciation sequence where the target pronunciation is located from the pronunciation sequence corresponding to the audio clip; where the first pronunciation sequence is the same as the second pronunciation sequence after conversion of the pinyin sequence.

[0169] Exemplarily, when the target pronunciation is "Bao", the first pronunciation sequence can be "Zhang, Yu, Bao", the pinyin sequence corresponding to the polyphonic word phrase "Octo Star" is "Zhang, Yu, Bao", and its corresponding second pronunciation sequence is also "Zhang, Yu, Bao".

[0170] Step D12: Update the matching probability of the matched candidate pinyin based on the recognition probability of each pronunciation in the first pronunciation sequence, and obtain the updated matching probability.

[0171] Here, the average recognition probability of multiple pronunciations in the first pronunciation sequence can be used to weight and update the matching probability of the matched candidate pinyin, and the final matching probability of the candidate pinyin can be obtained.

[0172] For example: The predicted P_Y("bao"|"堡") = 0.829 for the matched candidate pinyin. The three-pinyin combination with one character before and after the context is "Zhang, Yu, Bao". In the pronunciation sequence output by ASR, it is considered that the candidate pinyin "bao" matches the pronunciation sequence. Then, the probability of the "bao" pinyin of the polyphonic character "堡" in the subtitle is weighted, and P_YF("bao"|"堡") = min(1.0, 0.829 + avg_ASR(Zhang|0.76, Yu|0.65, Bao|0.79)) = 1.0, where avg_ASR(Zhang|0.76, Yu|0.65, Bao|0.79) represents the average value of the recognition probabilities of these three pronunciations.

[0173] The following combines Figure 10 to introduce the flowchart of another subtitle phonetic annotation method of the present application embodiment.

[0174] The subtitle phonetic annotation method of the embodiment of the present application can be executed independently by the terminal device or the server, or jointly executed by the terminal device and the server. The following takes the server execution as an example for illustration. Exemplarily, as Figure 10 shown, the subtitle phonetic annotation method of the embodiment of the present application includes the following steps S1001 - S1007:

[0175] Step S1001: The server phonetically annotates the single characters in the subtitle text.

[0176] Among them, by querying the above-mentioned association relationship between characters and pinyin, a pinyin corresponding to the single character can be obtained, and then the single character can be labeled with the pinyin.

[0177] Step S1002: The server determines whether the subtitle text contains polyphonic characters. If it contains, steps S1003 and S1004 are continued to be executed. If it does not contain, step S1007 is executed.

[0178] Step S1003: The server predicts the matching probability of multiple candidate pinyins of the polyphonic character based on the subtitle text and the audio segment corresponding to the subtitle text through the polyphonic character prediction model.

[0179] In this process, a polyphonic character prediction model is used to predict multiple candidate pinyin for polyphonic characters to obtain the matching probability of multiple candidate pinyin. For details, please refer to the above embodiments of this application, which will not be repeated here.

[0180] In step S1004, the server obtains the pronunciation sequence of the audio segment corresponding to the subtitle text through the ASR model.

[0181] Step S1005: The server updates the prediction results of the polyphonic character phonetic model based on the pronunciation sequence to obtain the updated prediction results.

[0182] In this step, the process of updating the prediction results of the phonetic annotation model for polyphonic characters based on the pronunciation sequence is described in the above embodiments of this application and will not be repeated here.

[0183] Step S1006: Based on the updated prediction results, the server obtains the target pinyin for the polyphonic characters and annotates the polyphonic characters with pinyin.

[0184] The updated prediction results include the matching probabilities of multiple candidate pinyin for polyphonic characters. The candidate pinyin with the highest matching probability can be selected as the target pinyin from among the multiple candidate pinyin.

[0185] In step S1007, the server completes the phonetic annotation of the subtitle text.

[0186] It should be noted that the execution order of the above steps is not limited in the embodiments of this application. For example, the order of steps S1003 and S1004 is not limited.

[0187] In this embodiment, when annotating the subtitle text in a video with phonetic symbols, the subtitle text is combined with the corresponding audio segment. Through multi-dimensional understanding of the subtitle text and audio segment, specifically by first comprehensively modeling the subtitle text and audio segment using a polyphonic character phonetic annotation model, and then combining the model output with the pronunciation sequence of the audio segment, the target pinyin of the polyphonic characters in the subtitle text is predicted more accurately. This not only makes the target pinyin more consistent with the context of the subtitle text but also makes it more consistent with the pronunciation context of the video segment in which the subtitle text is located. This improves the accuracy of the phonetic annotation of the subtitle text, enabling users to better understand the video content and enhancing the product experience of the video platform.

[0188] The following is combined Figure 11 The logical diagram of the subtitle phonetic annotation method of the present application embodiment is introduced below; the subtitle phonetic annotation method can be executed by the terminal device or the server alone, or by the terminal device and the server together. The following description takes the server execution as an example.

[0189] like Figure 11As shown in the figure, when the server phonetically annotates the subtitle text "Look, the Octopus Fort is repaired", it phonetically annotates the single characters "Look", "Zhang", "Yu", "Xiu", "Li", "Hao", and obtains "Kan", "Zhang", "Yu", "Xiu", "Li", "Hao" respectively. Further, the polyphonic character "Fort" is input into the polyphonic character phonetic annotation model, and the matching probabilities of the candidate pinyin "Bao", "Pu", "Bu" of "Fort" are output; the pronunciation sequence of the audio segment corresponding to the subtitle text is used to update the matching probability of "Bao", and finally the target pinyin of "Fort" is obtained as "Bao"; the polyphonic character "Le" is input into the polyphonic character phonetic annotation model, and the matching probabilities of the candidate pinyin "Le", "Liao" of "Le" are output, and the pronunciation sequence of the audio segment corresponding to the subtitle text is used to update the matching probability of "Le", and finally the target pinyin of "Le" is obtained as "Le". For the phonetic annotation effect of the above subtitle text, see Figure 12A as shown in the figure.

[0190] It should be noted that the subtitle phonetic annotation method in the embodiments of the present application can be applied not only to the above video playback scenario with subtitles, but also to any other playback scenario that includes subtitles and the audio corresponding to the subtitles. For example, if the recorded video includes a presentation (Microsoft Office PowerPoint, PPT) and the audio corresponding to the text in the PPT, the subtitle phonetic annotation method in the embodiments of the present application can be used to phonetically annotate the text in the PPT.

[0191] As Figure 12B shown in the figure, in the played PPT, the text "Spring is here, the weather is warm" is displayed. The text is phonetically annotated by the subtitle phonetic annotation method in the embodiments of the present application, and "Chun Tian Lai Le, Tian Qi Nuan HuoLe" is obtained. The correct pinyin of the polyphonic character "Huo" is "Huo", and the correct pinyin of "Le" is "Le".

[0192] Based on the same inventive concept as the above method embodiments of the present application, an subtitle phonetic annotation device is also provided in the embodiments of the present application. The principle of the device to solve the problem is similar to the method in the above embodiments. Therefore, the implementation of the device can refer to the implementation of the above method, and the repeated parts will not be elaborated.

[0193] As Figure 13 shown in the figure, an subtitle phonetic annotation device is provided in the embodiments of the present application, including an acquisition module 131, a text feature extraction module 132, an audio feature extraction module 133, and a polyphonic character annotation module 134.

[0194] The acquisition module 131 is used to acquire multiple candidate pinyin respectively associated with at least one polyphonic character included in the subtitle text;

[0195] The text feature extraction module 132 is used to extract semantic features from the subtitle text, obtain the text semantic features corresponding to the subtitle text, and obtain the semantic features corresponding to at least one polyphonic character based on the text semantic features.

[0196] The audio feature extraction module 133 is used to extract audio features from the audio segments corresponding to the subtitle text and obtain the audio features corresponding to the audio segments.

[0197] The polyphonic character annotation module 134 is used to perform the following operations for at least one polyphonic character: based on the semantic features and audio features of a polyphonic character, determine the matching probability of each of the multiple candidate pinyin associated with the polyphonic character, and select the target pinyin from the multiple candidate pinyin based on the multiple matching probabilities to annotate the polyphonic character.

[0198] Optionally, when determining the matching probability of each of the multiple candidate pinyin associated with a polyphonic character based on its semantic and audio features, the polyphonic character phonetic annotation module 134 is also used for:

[0199] An attention mechanism is used to process the semantic and audio features of a polyphonic character to obtain interactive features;

[0200] Based on the semantic features, audio features, and interaction features of a polyphonic character, the matching probability of each of the multiple candidate pinyin associated with a polyphonic character is determined.

[0201] Optional, such as Figure 14 As shown, the device also includes:

[0202] The speech recognition module 135 is used to perform speech recognition on the audio segment corresponding to the subtitle text, obtain the pronunciation sequence corresponding to the audio segment and the recognition probability of each pronunciation, and obtain the target pronunciation corresponding to a polyphonic character from the pronunciation sequence;

[0203] When selecting the target pinyin from multiple candidate pinyin based on multiple matching probabilities, and annotating a polyphonic character, the polyphonic character annotation module 134 is also used for:

[0204] If among multiple candidate pinyin, there is a candidate pinyin that matches the target pronunciation, then based on the recognition probability of the target pronunciation, the matching probability of the matching candidate pinyin is updated to obtain the updated matching probability;

[0205] From multiple candidate pinyin, select the candidate pinyin whose matching probability satisfies the probability condition as the target pinyin, and mark a polyphonic character.

[0206] Optionally, the device further includes a matching module 136 for determining candidate pinyin that matches the target pronunciation from a plurality of candidate pinyin in the following manner:

[0207] For multiple candidate pinyin, perform the following operations respectively:

[0208] Select phrases containing a polyphonic character from the subtitle text and determine the corresponding pinyin sequence. The pinyin sequence contains a candidate pinyin corresponding to a polyphonic character.

[0209] If the pinyin sequence matches the pronunciation sequence corresponding to the audio segment, then a candidate pinyin is selected as the candidate pinyin that matches the target pronunciation.

[0210] Optionally, based on the recognition probability of the target pronunciation, the matching probability of the candidate pinyin is updated. When obtaining the updated matching probability, the polyphonic character phonetic notation module 134 is also used for:

[0211] From the pronunciation sequences corresponding to the audio segment, select the first pronunciation sequence containing the target pronunciation; wherein, the first pronunciation sequence is the same as the second pronunciation sequence after the pinyin sequence is converted;

[0212] Based on the recognition probabilities of multiple pronunciations in the first pronunciation sequence, the matching probabilities of the candidate pinyin are updated to obtain the updated matching probabilities.

[0213] Optionally, the text feature extraction module 132 is also used for:

[0214] Input the subtitle text into a pre-trained language model to obtain the text's semantic features;

[0215] For at least one polyphonic character, perform the following operations: Based on the position information of a polyphonic character in the subtitle text, select the semantic features corresponding to the position information from the text semantic features.

[0216] Optionally, the audio feature extraction module 133 is also used for:

[0217] Perform Fourier transform on the audio segment corresponding to the subtitle text to obtain the spectrum of the audio segment;

[0218] Multiple audio frames obtained based on the spectrogram are input into the audio feature extraction model to obtain the audio frame features corresponding to each audio frame.

[0219] The features of multiple audio frames obtained are used as the audio features corresponding to the audio segments.

[0220] Optionally, the audio feature extraction model includes a deep representation module and an attention-based feature extraction module;

[0221] When multiple audio frames obtained based on the spectrogram are input into the audio feature extraction model to obtain the audio frame features corresponding to each audio frame, the audio feature extraction module 133 is also used for:

[0222] Multiple audio frames obtained based on the spectrogram are input into the depth representation module to obtain the intermediate audio frame features corresponding to each of the multiple audio frames.

[0223] The obtained intermediate audio frame features are input into the attention-based feature extraction module to obtain the audio frame features corresponding to each audio frame.

[0224] Optionally, the subtitle text also contains at least one monosyllabic character, and the device further includes a monosyllabic character annotation module 137 for:

[0225] Get the pinyin associated with at least one monosyllabic character;

[0226] Perform the following operations for at least one monosyllabic character: use a pinyin associated with a monosyllabic character to annotate the monosyllabic character.

[0227] In this embodiment, when annotating the subtitle text in a video with pinyin, the subtitle text is combined with the corresponding audio segment. Through multi-dimensional understanding of the subtitle text and audio segment, specifically by first comprehensively modeling the subtitle text and audio segment using a polyphonic character pinyin model, and then combining the model output with the pronunciation sequence of the audio segment, the target pinyin of the polyphonic characters in the subtitle text is predicted more accurately. This makes the target pinyin more consistent with the context of the subtitle text and the pronunciation context of the video segment in which the subtitle text is located. This improves the accuracy of the pinyin annotation, enabling users to better understand the video content and enhancing the product experience of the video platform.

[0228] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0229] Regarding the apparatus in the above embodiments, the specific execution methods of each module have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0230] Those skilled in the art will understand that various aspects of this application can be implemented as devices, methods, or computer program products, with each part described separately as a module according to its function. Of course, in implementing this application, the functions of each module can be implemented in one or more software or hardware components. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0231] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. The principle of this electronic device in solving the problem is similar to that of the above-described method. Therefore, the implementation of this electronic device can refer to the implementation of the above-described method, and repeated details will not be described again.

[0232] See Figure 15 As shown, the electronic device 150 may include at least a processor 151 and a memory 152. The memory 152 stores a computer program, which, when executed by the processor 151, causes the processor 151 to perform the steps in any of the aforementioned subtitle phonetic annotation methods.

[0233] In an exemplary embodiment, this application also provides a storage medium including a computer program, such as a memory 152 including a computer program, which can be executed by a processor 151 of an electronic device 150 to complete the above-described subtitle phonetic annotation method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0234] The following reference Figure 16 To describe an electronic device 160 according to this embodiment of the present application. Figure 16 The electronic device 160 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0235] like Figure 16 The electronic device 160 is manifested in the form of a general electronic device. The components of the electronic device 160 may include, but are not limited to: at least one processing unit 161, at least one storage unit 162, and a bus 163 connecting different system components (including storage unit 162 and processing unit 161).

[0236] Bus 163 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus architectures.

[0237] Storage unit 162 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1621 and / or cache memory 1622, and may further include read-only memory (ROM) 1623.

[0238] Storage unit 162 may also include a program / utility 1625 having a set (at least one) of program modules 1624, such program modules 1624 including but not limited to: operating system, one or more application programs, other program modules and program data, each of these examples or some combination of these may include an implementation of a network environment.

[0239] Electronic device 160 can also communicate with one or more external devices 164 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 160, and / or with any device that enables electronic device 160 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 165. Furthermore, electronic device 160 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 166. As shown, network adapter 166 communicates with other modules used in electronic device 160 via bus 163. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 160, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0240] Based on the same inventive concept as the above-described method embodiments, this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the steps of any of the above-described subtitle phonetic annotation methods.

[0241] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0242] The computer program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the computer program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0243] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0244] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0245] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0246] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A subtitle annotation method, characterized by, include: Obtain multiple candidate pinyin for each of the at least one polyphonic character contained in the subtitle text; Semantic features are extracted from the subtitle text to obtain the text semantic features corresponding to the subtitle text, and the semantic features corresponding to each of the at least one polyphonic character are obtained based on the text semantic features; Audio features are extracted from the audio segment corresponding to the subtitle text to obtain the audio features corresponding to the audio segment; For each of the at least one polyphonic character, the following operations are performed: an attention mechanism is used to process the semantic features and audio features of the polyphonic character to obtain interactive features; based on the semantic features, audio features, and interactive features of the polyphonic character, the matching probability of each of the multiple candidate pinyin associated with the polyphonic character is determined; and based on the multiple matching probabilities, the target pinyin is selected from the multiple candidate pinyin to annotate the polyphonic character.

2. The method of claim 1, wherein, Before selecting the target pinyin from the multiple candidate pinyin based on multiple matching probabilities and labeling the polyphonic character, the method further includes: Speech recognition is performed on the audio segment corresponding to the subtitle text to obtain the pronunciation sequence corresponding to the audio segment and the recognition probability of each pronunciation, and the target pronunciation corresponding to the polyphonic character is obtained from the pronunciation sequence; The step of selecting the target pinyin from the multiple candidate pinyin based on multiple matching probabilities and annotating the polyphonic character includes: If there is a candidate pinyin among the multiple candidate pinyin that matches the target pronunciation, then the matching probability of the matching candidate pinyin is updated based on the recognition probability of the target pronunciation to obtain the updated matching probability; From the multiple candidate pinyin, the candidate pinyin whose matching probability satisfies the probability condition is selected as the target pinyin, and the polyphonic character is marked.

3. The method according to claim 2, characterized in that, The candidate pinyin that matches the target pronunciation among the plurality of candidate pinyin is determined in the following way: For each of the multiple candidate pinyin, perform the following operations: Select a phrase containing the polyphonic character from the subtitle text, and determine the pinyin sequence corresponding to the phrase, wherein the pinyin sequence contains a candidate pinyin corresponding to the polyphonic character; If the pinyin sequence matches the pronunciation sequence corresponding to the audio segment, then the candidate pinyin is used as a candidate pinyin that matches the target pronunciation.

4. The method according to claim 3, characterized in that, The step of updating the matching probability of the candidate pinyin based on the recognition probability of the target pronunciation to obtain the updated matching probability includes: From the pronunciation sequences corresponding to the audio segment, select the first pronunciation sequence containing the target pronunciation; wherein, the first pronunciation sequence is the same as the second pronunciation sequence after the pinyin sequence is converted; Based on the recognition probabilities of multiple pronunciations in the first pronunciation sequence, the matching probabilities of the candidate pinyin are updated to obtain the updated matching probabilities.

5. The method according to claim 1, characterized in that, The step of extracting semantic features from the subtitle text to obtain the text semantic features corresponding to the subtitle text, and obtaining the semantic features corresponding to each of the at least one polyphonic character based on the text semantic features, includes: The subtitle text is input into a pre-trained language model to obtain the semantic features of the text; For each of the at least one polyphonic characters, perform the following operations: based on the position information of the polyphonic character in the subtitle text, select the semantic feature corresponding to the position information from the text semantic features.

6. The method according to claim 1, characterized in that, The step of extracting audio features from the audio segment corresponding to the subtitle text to obtain the audio features corresponding to the audio segment includes: The audio segment corresponding to the subtitle text is subjected to Fourier transform processing to obtain the spectrum diagram corresponding to the audio segment; The multiple audio frames obtained based on the spectrogram are input into the audio feature extraction model to obtain the audio frame features corresponding to each of the multiple audio frames. The obtained features of multiple audio frames are used as the audio features corresponding to the audio segment.

7. The method according to claim 6, characterized in that, The audio feature extraction model includes a deep representation module and an attention-based feature extraction module; The step of inputting multiple audio frames obtained based on the spectrogram into an audio feature extraction model to obtain the audio frame features corresponding to each of the multiple audio frames includes: The multiple audio frames obtained based on the spectrogram are input into the depth representation module to obtain the intermediate audio frame features corresponding to each of the multiple audio frames. The obtained intermediate audio frame features are input into the attention-based feature extraction module to obtain the audio frame features corresponding to each of the multiple audio frames.

8. The method according to claim 1, characterized in that, The subtitle text also contains at least one monosyllabic character. Before extracting semantic features from the subtitle text to obtain the corresponding text semantic features, the method further includes: Obtain a pinyin associated with each of the at least one monosyllabic character; For each of the at least one monosyllabic characters, perform the following operations: use a pinyin associated with a monosyllabic character to annotate the monosyllabic character.

9. A phonetic subtitle device, characterized in that, include: The acquisition module is used to acquire multiple candidate pinyin associated with each of the at least one polyphonic character contained in the subtitle text; The text feature extraction module is used to extract semantic features from the subtitle text, obtain the text semantic features corresponding to the subtitle text, and obtain the semantic features corresponding to each of the at least one polyphonic character based on the text semantic features; The audio feature extraction module is used to extract audio features from the audio segment corresponding to the subtitle text to obtain the audio features corresponding to the audio segment. The polyphonic character annotation module is used to perform the following operations for the at least one polyphonic character: process the semantic features and audio features of a polyphonic character using an attention mechanism to obtain interaction features; determine the matching probability of each of the multiple candidate pinyin associated with the polyphonic character based on the semantic features, audio features, and interaction features; and select the target pinyin from the multiple candidate pinyin based on the multiple matching probabilities to annotate the polyphonic character.

10. The apparatus according to claim 9, characterized in that, Also includes: The speech recognition module is used to perform speech recognition on the audio segment corresponding to the subtitle text, obtain the pronunciation sequence corresponding to the audio segment and the recognition probability of each pronunciation, and obtain the target pronunciation corresponding to the polyphonic character from the pronunciation sequence; When selecting the target pinyin from the multiple candidate pinyin based on multiple matching probabilities and annotating the polyphonic character, the polyphonic character annotation module is further used for: If there is a candidate pinyin among the multiple candidate pinyin that matches the target pronunciation, then the matching probability of the matching candidate pinyin is updated based on the recognition probability of the target pronunciation to obtain the updated matching probability; From the multiple candidate pinyin, the candidate pinyin whose matching probability satisfies the probability condition is selected as the target pinyin, and the polyphonic character is marked.

11. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 8.

12. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 8.

13. A computer program product, characterized in that, It includes a computer program stored in a computer-readable storage medium; when the processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 8.

Images