Text display method and device, electronic equipment and readable storage medium

Abstract

The application discloses a text display method and device, electronic equipment and readable storage medium, and belongs to the technical field of video processing. The method comprises the following steps: acquiring audio information of a first video in the case of playing the first video; acquiring first pitch feature of background voice information and first source text information in the audio information; acquiring a pitch feature matrix corresponding to the first pitch feature through an annotation embedding layer, and acquiring a text feature matrix corresponding to the first source text information through a first attention network; fusing the pitch feature matrix and the text feature matrix through a second attention network to obtain a fusion feature matrix, and determining first translation text information of the first source text information based on the fusion feature matrix through a full connection layer; determining second translation text information of subject voice information in the audio information; and displaying the first translation text information and the second translation text information.

Description

Technical Field

[0001] This application belongs to the field of video processing technology, and more specifically, relates to a text display method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the rapid development of machine translation technology, it has evolved from basic text translation to support multimodal translation, including image translation, speech translation, and video translation. As a medium that simultaneously contains images and audio, current video translation technologies primarily focus on translating the audio content into subtitles.

[0003] While mobile translation apps currently offer a wealth of translation functions, including but not limited to text translation, image translation, real-time translation, voice translation, and AI (Artificial Intelligence) subtitle translation, users still have to manually pause the video, open the translation app, select the appropriate translation function, and then return to the current video to translate whenever they need to. This process is cumbersome and disrupts the smooth viewing experience. Furthermore, the audio in videos is often complex; directly translating it as subtitles and displaying it mixed with regular subtitles results in poor translation quality, negatively impacting the user's viewing experience. Summary of the Invention

[0004] The purpose of this application is to provide a text display method, device, electronic device, and readable storage medium that can solve the problems of cumbersome and poor translation quality when watching videos.

[0005] In a first aspect, embodiments of this application provide a text display method, the method comprising:

[0006] When playing the first video, obtain the audio information of the first video;

[0007] Obtain the first pitch feature and the first source text information of the background speech information in the audio information;

[0008] The tone feature matrix corresponding to the first tone feature is obtained through the annotation embedding layer, and the text feature matrix corresponding to the first source text information is obtained through the first attention network.

[0009] The pitch feature matrix and the text feature matrix are fused using a second attention network to obtain a fused feature matrix, and the first translated text information of the first source text information is determined based on the fused feature matrix using a fully connected layer.

[0010] Determine the second translated text information of the main speech information in the audio information;

[0011] Display the first translated text information and the second translated text information.

[0012] Secondly, embodiments of this application provide a text display device, the device comprising:

[0013] The first acquisition module is used to acquire the audio information of the first video while the first video is playing;

[0014] The second acquisition module is used to acquire the first pitch feature and the first source text information of the background speech information in the audio information;

[0015] The third acquisition module is used to acquire the pitch feature matrix corresponding to the first pitch feature through the annotation embedding layer, and to acquire the text feature matrix corresponding to the first source text information through the first attention network.

[0016] The first determining module is used to fuse the pitch feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix, and to determine the first translated text information of the first source text information based on the fused feature matrix through a fully connected layer;

[0017] The second determining module is used to determine the second translated text information of the main speech information in the audio information;

[0018] The display module is used to display the first translated text information and the second translated text information.

[0019] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0020] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0021] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0022] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0023] In this embodiment, when playing a first video, the electronic device acquires the audio information of the first video and automatically obtains the first pitch feature of the background speech information and the first source text information in the audio information. It then obtains the pitch feature matrix corresponding to the first pitch feature through an annotation embedding layer, and the text feature matrix corresponding to the first source text information through a first attention network. Finally, it fuses the pitch feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix. Based on the fused feature matrix, a fully connected layer determines the first translated text information of the first source text information. Simultaneously, it also determines the second translated text information of the main speech information in the audio information, and then displays the first and second translated text information. That is, when playing the first video, the electronic device can translate the audio information in the first video without switching to a translation application. Furthermore, it can distinguish between background speech information and main speech information, and display the first translated text information of the background speech information and the second translated text information of the main speech information. Moreover, it adaptively enhances the translation of the background speech information based on pitch features, thereby achieving high-quality translation and improving the user's viewing experience. Attached Figure Description

[0024] Figure 1 A flowchart illustrating the text display method provided in this application embodiment;

[0025] Figure 2 One of the schematic diagrams showing a video frame provided in an embodiment of this application;

[0026] Figure 3 A musical score diagram provided for an embodiment of this application;

[0027] Figure 4 A schematic diagram illustrating the training of the background speech recognition model provided in this application embodiment;

[0028] Figure 5 This is a schematic diagram of the structure of the text recognition model provided in the embodiments of this application;

[0029] Figure 6 A second schematic diagram showing a video frame provided in an embodiment of this application;

[0030] Figure 7 The third schematic diagram showing a video frame provided in an embodiment of this application;

[0031] Figure 8 A flowchart illustrating a text display method as an example of this application;

[0032] Figure 9 This is a schematic diagram of the structure of the text display device provided in the embodiments of this application;

[0033] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0034] Figure 11 A schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0036] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0037] The text display method provided in this application embodiment can be implemented by a text display device provided in this application embodiment, or an electronic device integrating the text display device, wherein the text display device can be implemented in hardware or software.

[0038] The text display method provided in this application, when playing a first video, can automatically distinguish between background audio information and main audio information, and translate the background audio information and main audio information separately to display them as two different types of subtitle information. Simultaneously, when translating the background audio information, it enhances the translation based on the tonal characteristics of the background audio information, thereby improving the translation quality. Translation, in this context, refers to the technique of converting a source language into a target language.

[0039] The text display method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0040] like Figure 1 As shown, this is a text display method provided in an embodiment of this application, which is applied to an electronic device. Figure 1As shown, the text display method may include the following steps 1100 to 1600, which will be described in detail below.

[0041] Step 1100: While playing the first video, obtain the audio information of the first video.

[0042] The first video is the video that the user is playing on their electronic device. This video can be a short video, a TV series, a movie, etc. For example, the first video can be a video played by the user through a browser interface, a video application, or a social media application.

[0043] In this embodiment, when the user is watching the first video, the electronic device will automatically extract audio segments from the first video.

[0044] Step 1200: Obtain the first pitch feature and the first source text information of the background speech information in the audio information.

[0045] Understandably, subtitles help users understand the audio content of films and television dramas. In this embodiment, the source language and target language can be set by the user. The source language is a commonly used language set by the user and does not require translation. The target language is a less commonly used language set by the user and is the language that needs to be translated. For example, Chinese is used as the commonly used language set by the user, and English is used as the less commonly used language that needs to be translated. The following examples will all use Chinese as the source language and English as the target language.

[0046] In this embodiment, since the audio information in the first video is usually quite complex, it may contain multiple voice messages, and some of these voice messages may be background music. If it is directly translated and used as subtitles, mixed with regular subtitles, the translation quality will be reduced, affecting the user's viewing experience. Therefore, it is necessary to classify and identify the audio information. Here, in this embodiment, after performing step 1100 to obtain the audio information of the first video, it is necessary to further identify whether the audio information includes background voice information. If it includes background voice information, then step 1200 is performed to obtain the first pitch feature and the first source text information of the background voice information in the audio information. Therefore, the text display method of this embodiment further includes the following steps 2100 to 2400:

[0047] Step 2100: Perform speech recognition on the audio information to obtain the target source text information.

[0048] In this step 2100, the electronic device can perform Automatic Speech Recognition (ASR) on the captured audio segment to convert the audio information into text information as the target source text information.

[0049] Step 2200: Extract the target tone features of the target source text information.

[0050] Step 2300: Based on the target pitch features, the target source text information, and the preset background speech recognition model, determine whether the audio information includes background speech information.

[0051] The background speech recognition model is used to identify background speech information in audio information. The input of the background speech recognition model is the text information to be predicted and the corresponding pitch information of the text information. The output of the background speech recognition model is a label indicating whether the audio information includes background speech information.

[0052] In step 2300, the target pitch features and target source text information are input into the background speech recognition model, which can then identify whether the audio information includes background speech information, i.e., whether the audio segment includes a song. In this embodiment, the background speech recognition model can be trained based on training samples first, and then the background speech information in the audio information can be identified based on the background speech recognition model. Here, the text display method of this embodiment further includes the following steps 3100 to 3300:

[0053] Step 3100: Obtain the first training sample set.

[0054] The first training sample set includes a first training sample and a second training sample. The first training sample includes lyrics samples, a first pitch sample corresponding to the lyrics samples, and a tag indicating that the lyrics samples are songs. The second training sample includes the lyrics samples, a second pitch sample corresponding to the lyrics samples, and a tag indicating that the lyrics samples are not songs. The first pitch sample is the pitch sample of the lyrics in the musical score, and the second pitch sample is the pitch sample when the data collector reads the lyrics based on the language of the lyrics.

[0055] Understandably, referring to Figure 3 The lyrics and musical notes in a song follow a certain rhythm and pattern; however, our everyday speech is more straightforward, with significant differences in pitch. Therefore, electronic devices can first acquire the lyrics and pitch from the song's score, and then data collectors can read the lyrics aloud in Mandarin. For example, regarding... Figure 3The lyrics and tones of the shown sheet music are acquired, namely "Happy Birthday to You" and "556517", then the data collector reads "Happy Birthday to You" in Mandarin, and the speech is converted to the tone "136333".

[0056] Exemplarily, the lyrics and the tones are combined to serve as the semantic feature and the tone feature of one training sample respectively. Here, the lyric sample and the tone sample in the sheet music are taken as one positive training sample, for example, ("Happy Birthday to You", "556517", 1), and the lyric sample and the tone sample obtained by the data collector reading the lyrics in Mandarin are taken as one negative training sample, for example, ("Happy Birthday to You", "136333", 0), so as to improve the prediction accuracy of the music training model.

[0057] Step 3200: A background speech recognition model is obtained through training according to the first training sample set.

[0058] Taking the two training samples above as an example, the lyric character string and the tone character string are respectively subjected to word segmentation processing to obtain a text token list and a note token list. Wherein, the text token list ["Wish you", "birthday", "happiness"] and the note token list ["5", "5", "6", "5", "1", "7"] are used as input data for background speech recognition, and 1 is used as the output label of the background speech recognition model. The text token list ["Wish you", "birthday", "happiness"] and the note token list ["1", "3", "6", "3", "3", "3"] are used as input data of the background speech recognition model, and 0 is used as the output label of the background speech recognition model, to perform training of the background speech recognition model.

[0059] With reference to Figure 4 , the text input and the note input pass through the word embedding layer and the note embedding layer of the background speech recognition model respectively to obtain a text feature matrix Matrix word and a note feature matrix Matrix note . Wherein, the text feature matrix Matrix word represents the feature information of each word token in the lyric sample, and the note feature matrix Matrix note represents the feature information of each note token in the tone sample. The two matrices are spliced through the splicing function (concat) of the background speech recognition model to obtain Vector word+note , then the Vector word+noteThe background speech recognition model performs feature calculation and label prediction through the Convolutional Neural Networks (CNN) layer, and calculates the loss based on the predicted labels and the real labels to update the model parameters of the background speech recognition model, thereby obtaining a trained background speech recognition model.

[0060] Based on steps 3100 to 3200 above, a background speech recognition model can be trained based on the first training sample set, and then the background speech recognition model can be used to recognize background speech information in audio information, thereby improving the accuracy of background speech information recognition.

[0061] Step 2400: If the audio information includes background speech information, obtain the first pitch feature and the first source text information of the background speech information in the audio information.

[0062] Based on steps 2100 to 2400 above, it can distinguish between the vocal parts of the song and the normal dialogue parts, and more accurately identify the scenes that need to be translated.

[0063] In this embodiment, when the audio information includes background speech information, the electronic device will further perform speech recognition on the background speech information to obtain the first source text information of the background speech information, and identify the first tone feature of the background speech information. Then, based on the tone feature of the background speech information and the source text information, the translated text information corresponding to the background speech information is obtained.

[0064] Step 1300: Obtain the pitch feature matrix corresponding to the first pitch feature through the annotation embedding layer, and obtain the text feature matrix corresponding to the first source text information through the first attention network.

[0065] In this embodiment, the first tone feature and the first source text information can be predicted based on a text translation model to obtain the first translated text information. That is, the input of the text translation model is the first tone feature and the first source text information, and the output of the text translation model is the first translated text information.

[0066] In this embodiment, refer to Figure 5 The text translation model includes an annotation embedding layer, a first word embedding layer, a second word embedding layer, an encoder, and a decoder. The decoder consists of a first attention network, a second attention network, and a fully connected layer.

[0067] It is important to note that when translating the first source text information and the first tone feature using the text translation model, it is processed on a word token basis. During the first translation, a start identifier [CLS] is used as input. After translating the first word token of the target language, the translated first word token is used as input to translate the second word token, and so on.

[0068] During the initial translation, the first word token from the first source text can be input into the first attention network through the first word embedding layer and the encoder, while the starting identifier [CLS] can be input into the first attention network through the second word embedding layer. The first attention network then calculates the text feature matrix between the first word token and "[CLS]" from the first source text. Here, dim represents the hidden layer dimension. Furthermore, the first word token corresponding to the first tone feature is input into the annotation embedding layer to obtain the tone feature matrix.

[0069] Step 1400: The pitch feature matrix and the text feature matrix are fused through a second attention network to obtain a fused feature matrix, and the first translated text information of the first source text information is determined based on the fused feature matrix through a fully connected layer.

[0070] In this embodiment, the process of fusing the pitch feature matrix and the text feature matrix using a second attention network in step 1400 to obtain the fused feature matrix may further include the following steps 1410 to 1460:

[0071] Step 1410: Multiply the text feature matrix and the first parameter matrix of the fully connected layer using the second attention network to obtain the text feature transformation matrix.

[0072] In step 1410, the text feature matrix is... The first parameter matrix of the fully connected layer Matrix multiplication is performed to extract features, resulting in the text feature transformation matrix.

[0073] Step 1420: Multiply the pitch feature matrix and the second parameter matrix of the fully connected layer to obtain the first pitch feature transformation matrix.

[0074] In step 1420, the pitch feature matrix is ​​combined with the second parameter matrix of the fully connected layer. Matrix multiplication is performed to extract features, resulting in the first tone feature transformation matrix.

[0075]

[0076] Step 1430: Multiply the pitch feature matrix and the third parameter matrix of the fully connected layer to obtain the second pitch feature transformation matrix.

[0077] In step 1430, the pitch feature matrix is ​​compared with the third parameter matrix of the fully connected layer. Matrix multiplication is performed to extract features, resulting in the second tone feature transformation matrix.

[0078]

[0079] Step 1440: Multiply the text feature transformation matrix and the first tone feature transformation matrix to obtain the attention matrix.

[0080] In step 1440, the text feature transformation matrix and the first pitch feature transformation matrix are extracted into an attention network to obtain the attention matrix Attn. The values ​​in this attention matrix can represent the correlation between word tokens and corresponding note tokens in the target language, that is, the weight of the note tokens to word tokens in the target language.

[0081]

[0082] Step 1450: Multiply the attention matrix and the second pitch feature transformation matrix to obtain the fused feature matrix.

[0083] In step 1450, the attention matrix Attn is combined with the second tone feature matrix. Perform matrix multiplication to obtain the fused feature matrix. dim :

[0084]

[0085] Step 1460: The first translated text information of the first source text information is determined by the fully connected layer based on the fusion feature matrix.

[0086] In step 1460, the fused feature matrix will be... dimPerform general fully connected layer calculations and label prediction to predict the first word token in the target language corresponding to the first word token in the first source text information. It can be understood that, based on steps 1410 to 1460 above, different word tokens in the target language corresponding to different word tokens in the first source text information can be predicted, and the resulting different word tokens in the target language are sequentially combined to form the first translated text information.

[0087] In other words, it can input the pitch features of the background speech information and the source text information of the background speech information into a text translation model for translation, so as to obtain the translated text information corresponding to the background speech information. Here, the text display method of this embodiment may further include the step of training a text translation model, which may further include the following steps 4100 to 4200:

[0088] Step 4100: Obtain the second training sample set.

[0089] The second training sample set includes a third training sample, which includes lyrics samples, pitch samples of the lyrics samples, and translation text tags of the lyrics samples.

[0090] Understandably, considering the stylistic differences between lyric translation and regular text translation, translation enhancement training is needed for music lyrics to ensure the translation is closer to the original. Currently, mainstream text translation models generally use the Transformer model. Unlike traditional CNNs and Recurrent Neural Networks (RNNs), the Transformer network is entirely based on an attention mechanism. Utilizing this mechanism, it can efficiently model the semantic features of both the source and target languages, thus achieving high-quality translation. In contrast to model prediction, model training uses the entirety of the target language sentence's word tokens as input, while model prediction is performed on individual word tokens. Previously predicted target language word tokens are used as input for the next iteration, and this process is repeated cyclically.

[0091] To enhance the translation of lyrics, an attention mechanism is used to model the relationship between the pitch features in the song and the target language. The pitch features are integrated into the features of the target language, so that the Transformer model fully considers the pitch features when predicting the tokens of the target language, thereby assisting in the translation of the target language.

[0092] Step 4200: Train the text translation model based on the second training sample set.

[0093] Optionally, step 4200, which trains the text translation model based on the second training sample set, may further include the following steps 4110 to 4150:

[0094] Step 4110: Obtain the pitch feature matrix of the pitch sample through the injection embedding layer.

[0095] Reference Figure 5 By segmenting the pitch samples of the lyrics into words to obtain a list of musical note tokens, and then inputting this list into the annotation embedding layer, a pitch feature matrix can be obtained. Where srclen represents the number of note tokens in the pitch sample, and dim represents the size of the hidden layer dimension.

[0096] Step 4120: Obtain the text feature matrix between the translated text labels and the lyrics samples through the first attention network.

[0097] Reference Figure 5 The lyrics samples are segmented to obtain source text tokens, which are then input into the first attention network via a first word embedding layer and an encoder. Similarly, the translated text tags of the lyrics samples are segmented to obtain target language tokens, which are then input into the first attention network via a second word embedding layer. The first attention network then yields the text feature matrix between the translated text tags and the lyrics samples. Where trglen represents the number of tokens in the target language, and dim represents the size of the hidden layer dimension.

[0098] Step 4130: The pitch feature matrix is ​​integrated into the text feature matrix through the second attention network to obtain a fused feature matrix.

[0099] In step 4130, the Attention mechanism is used to... and Perform feature calculation, The tonal features of the Chinese The specific steps include the following:

[0100] Step 1: The second attention network combines the semantics of the source language and the semantics of the target language. With the parameter matrix of the fully connected layer Matrix multiplication is performed to extract features, resulting in a new feature matrix.

[0101]

[0102] Step 2: The pitch feature matrix is ​​processed through a second attention network. With the parameter matrix of the fully connected layer Matrix multiplication is performed to extract features, resulting in a new feature matrix. Then perform matrix transpose to obtain

[0103]

[0104]

[0105] Step 3, convert the pitch feature matrix With the parameter matrix of the fully connected layer Matrix multiplication is performed to extract features, resulting in a new feature matrix.

[0106]

[0107] Step 4 involves combining the feature matrix that incorporates both source language semantics and target language semantics. With pitch feature matrix Extract attention features to obtain the attention matrix Attn trglen*srclen The values ​​in this attention matrix can represent the correlation between each token in the target language and each note token, that is, the weight of the note token to the tokens in the target language:

[0108]

[0109] Step 5, apply the attention matrix Attn trglen*srclen With pitch feature matrix Perform matrix multiplication to obtain the matrix. trglen*dim :

[0110]

[0111] Step 4140: Determine the loss of the translation model based on the fused feature matrix through the fully connected layer.

[0112] Reference Figure 5 The feature matrix that incorporates the characteristics of musical notes trglen*dim Perform general fully connected layer computation and label prediction to determine the loss of the text translation model.

[0113] Step 4150: Adjust the model parameters of the translation model according to the loss to obtain the text translation model.

[0114] Based on steps 4110 to 4150 above, the attention mechanism is used to model the relationship between musical note features and the target language, integrating note features into the target language features. This allows the Transformer model to fully consider note features when predicting tokens in the target language, thereby assisting in the translation of the target language. Here, a background speech recognition model is used to determine if the audio information is music, and a text translation model is used to perform enhanced translation of the lyrics text identified as music, resulting in the translation.

[0115] Step 1500: Determine the second translated text information of the main speech information in the audio information.

[0116] The main audio information is the audio information of the characters speaking in the video. It can be understood that the audio information can include background audio information and main audio information. After identifying the background audio information, the main audio information can be determined based on the audio information and the background audio information.

[0117] In this embodiment, speech recognition can be performed on the main speech information in the audio information to obtain the text information corresponding to the main speech information, and the text information corresponding to the main speech information can be translated to obtain the corresponding translated text information.

[0118] Step 1600: Display the first translated text information and the second translated text information.

[0119] In this embodiment, video frames may be displayed on the first page and first translated text information and second translated text information may be displayed on the second page. Alternatively, video frames, first translated text information and second translated text information may be displayed simultaneously on the first page.

[0120] In this embodiment, when the translation result is output and displayed as subtitles, it is necessary to distinguish between the second translated text information and the first translated text information. (Refer to...) Figure 6 The bottom area of ​​video frame 21 displays the second translated text information 61 and the first translated text information 62, and the second translated text information 61 and the first translated text information 62 are displayed on different lines in the bottom area of ​​video frame 21, and the second translated text information 61 and the first translated text information 62 are displayed in different colors.

[0121] According to an embodiment of this application, when playing a first video, the electronic device acquires the audio information of the first video, and automatically acquires the first pitch feature of the background speech information and the first source text information in the audio information. It then acquires the pitch feature matrix corresponding to the first pitch feature through an annotation embedding layer, and the text feature matrix corresponding to the first source text information through a first attention network. Finally, it fuses the pitch feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix. Based on the fused feature matrix, a fully connected layer determines the first translated text information of the first source text information. Simultaneously, it also determines the second translated text information of the main speech information in the audio information, and then displays the first and second translated text information. That is, when playing the first video, the electronic device can translate the audio information in the first video without switching to a translation application. Furthermore, it can distinguish between background speech information and main speech information, and display the first translated text information of the background speech information and the second translated text information of the main speech information. Moreover, it adaptively enhances the translation of the background speech information based on pitch features, thereby achieving high-quality translation and improving the user's viewing experience.

[0122] In one embodiment, it can also automatically perform image text recognition on video frames and translate the recognized text. Here, when performing step 1100 above while playing the first video, the text display method of this embodiment may further include the following steps 5100 to 5400:

[0123] Step 5100: Obtain the video frame corresponding to the audio information.

[0124] In this embodiment, when the user is watching the first video, the electronic device will automatically extract video frames corresponding to the audio segments.

[0125] Step 5200: Perform image text recognition on the video frame to obtain third source text information.

[0126] Understandably, if the audio information in the first video needs translation, the video frames may also contain text information that needs translation. For example, English place names and object names often appear in film and television scenes, which can affect the user's understanding of the content. Audio translation can only solve the problem of understanding audio content, not the problem of understanding English in the video frame. Therefore, image translation of video frames is necessary.

[0127] In this embodiment, image text recognition (OCR) technology is used to recognize text information in video frames, obtaining n pieces of third-source text information, and then the language of these n pieces of third-source text information is identified. Here, n is an integer greater than 1.

[0128] Step 5300: If the language of the third source text information is the target language, translate the third source text information to obtain the third translated text information.

[0129] In this embodiment, the target language is obtained by language identification of the third source text information. If m out of n third source text information are in English, then each of these m third source text information is translated to obtain the third translated text information corresponding to each of the m third source text information. Here, m is an integer greater than 1, and n is greater than m.

[0130] Understandably, translating polysemous words has always been a challenge for translation technology. For example, the English word "on" is more likely to be translated as "on top of," but when "on" appears on a switch or appliance, it should be translated as "open." Similarly, the word "can" on a can is more likely to be translated as "can," but when "can" appears on a can, it should be translated as "canned food."

[0131] Here, when translating English in video footage, the translation process can be intervened based on the associated objects of the words. When a word or phrase is identified by OCR in a video frame, it is first determined whether it has multiple meanings that are easily confused. If there are multiple meanings that are easily confused, the associated objects of the text in the video frame are identified. If a certain meaning of a word or phrase is related to its associated objects in the video frame, then that meaning is selected as the final translation result.

[0132] To automatically determine the relevance of translated meanings to objects, statistical analysis can be performed on the bilingual data used to train the machine translation model to obtain the associated vocabulary and confidence levels of different translation results. Taking a Chinese-English bilingual data scenario as an example, for English words with multiple Chinese meanings, word alignment is used to find the corresponding Chinese words, which constitute the translation of the English word. Then, entity recognition is performed on the Chinese text, and the identified entities are the associated entity words of the translation. The confidence level of the English translation to the current Chinese text is increased based on the word frequency of the associated entity words.

[0133] Specifically, when the corresponding translation of the English word word_en in a Chinese sentence is word_zh_a, the associated entities entity_a, entity_b, and entity_c appear a_a, a_b, and a_c times respectively; and when the corresponding translation of the English word word_en in a Chinese sentence is word_zh_b, the associated entities entity_a, entity_b, and entity_c appear b_a, b_b, and b_c times respectively, then when the entity entity_a is used as the associated entity of the English word word_en, the confidence that word_en is translated as word_zh_a is a_a / (a_a+b_a). For example, through statistics on large-scale bilingual data, the following Table 1 is obtained:

[0134] Table 1 English Word and Corresponding Entity Information Table

[0135]

[0136] Optionally, translating the third source text information to obtain third translated text information in step 5300 may further include steps 5310 to 5330:

[0137] Step 5310: Translate the third source text information to obtain at least one piece of fourth translated text information corresponding to the third source text information.

[0138] Refer to Figure 7 , the third source text information "can" is translated, and at least the translated text information "ke yi" and "neng gou" corresponding to "can" are obtained.

[0139] Step 5320: Acquire preset mapping data.

[0140] The mapping data reflects the correspondence between different pieces of fourth translated text information and entity information, and the mapping data may be the data in Table 1 above.

[0141] Step 5330: Determine the third translated text information from the at least one piece of fourth translated text information according to the associated entity information of the third source text information in the video frame and the mapping data.

[0142] Refer to Figure 7 , the English word "can" appears on a can in a video frame. In the video frame, the word appears on a "food" object and is associated with the meaning of "can (tin)", so "can (tin)" is selected as the translation result of the word.

[0143] Step 5400: Display the third translated text information.

[0144] In this embodiment, m pieces of third source text information are translated to obtain m pieces of third translated text information, then the m pieces of third translated text information are displayed near the corresponding third source text information in the video frame, so that the user can conveniently read the translation result when watching the video. Reference Figure 6 , the translated Chinese text "can (the container)" is displayed near the corresponding original text "can" in the video frame, so that the user can conveniently read the translation result when watching the video.

[0145] According to the above steps 5100 to 5400, adaptive translation can be performed for the scene of the video picture content. First, by counting the picture features and the translation results of polysemous words, the association between different picture features and specific translation results is learned. When translating the picture images of the user's movie and television video scene, a translation result more suitable for the scene is provided based on the image features in the picture. That is, it can solve the problem that polysemous words are difficult to translate accurately in video picture translation based on the picture features of the video.

[0146] In one embodiment, after acquiring the audio information of the first video and the video frames corresponding to the audio information, the electronic device can automatically identify whether the audio information needs to be translated. Here, the text display method according to the embodiment of the present disclosure further includes the following steps 6100 to 6400:

[0147] Step 6100: In the case that the language of the audio information is the target language, perform image text recognition on the first preset area of the video frame to obtain a first recognition result.

[0148] It can be seen from the above analysis that the source language is the commonly used language set by the user and does not need to be translated. The target language is the non-commonly used language set by the user, which is the language that needs to be translated. Exemplarily, Chinese is taken as the commonly used language set by the user, and English is taken as the non-commonly used language that the user needs to translate. In the following, examples are given with the source language being Chinese and the target language being English.

[0149] The first preset area may be the bottom quarter area of the video frame. It can be understood that, with reference Figure 2 as shown, if the first video includes subtitle information 22, the subtitle information is usually located in the bottom quarter area 23 of the video frame 21.

[0150] In one specific embodiment, the electronic device can perform speech recognition on the captured audio segment to convert the audio information into text information, and further perform language identification on the text information to obtain the language of the text information, which is then used as the language of the audio information. For example, if the language of the audio information is Chinese, then audio translation is not required; if the language of the audio information is English, it is necessary to determine whether audio translation is required by combining video frames. In the case where the language of the audio information is English, refer to... Figure 2 OCR recognition can be performed on the bottom quarter region of video frame 21 to obtain the first recognition result.

[0151] Step 6200: If the first recognition result indicates that no text information has been recognized, it is determined that the audio information needs to be translated.

[0152] Translation is a technique that converts a source language into a target language. Continuing the example above, if the first recognition result indicates that no text information was recognized, it means that there is no subtitle information, indicating that the user has a need for audio information translation.

[0153] Step 6300: If the first recognition result indicates that text information has been recognized and the language of the text information is the target language, it is determined that the audio information needs to be translated.

[0154] Continuing the example above, if the first recognition result indicates the identification of text information, it is necessary to further identify the language of that text information. If the language of the text information is Chinese, it means that the video the user is watching has Chinese subtitles, and in this case, no audio translation is needed. If the language of the text information is only English, it means that the video the user is watching only has English subtitles and no Chinese subtitles, and the user needs to translate the audio information. In other words, if no text information is identified in the bottom quarter area of ​​the video frame, or if the identified text information is only in English, it means that it may be necessary to translate the audio information into Chinese and display it as subtitles in the bottom area of ​​the video frame.

[0155] Step 6400: If it is necessary to translate the audio information, obtain the first pitch feature of the background speech information and the first source text information in the audio information.

[0156] Based on steps 6100 to 6400 above, the electronic device can automatically identify whether the audio information needs to be translated by combining the video frame and its corresponding audio segment.

[0157] In one embodiment, users have different needs for translation strategies in different video scenarios. Here, after obtaining the video frames of the first video and the video frames corresponding to the audio information, the text display method of this embodiment may further include the following steps 7100 to 7200:

[0158] Step 7100: Determine the target scene type of the first video based on the video frames and the set scene recognition model.

[0159] The scene video model takes video frames as input and outputs the scene type based on the video. Scene types include, but are not limited to, film / television scenes and online class scenes.

[0160] In this embodiment, video frames of the first video can be input into the scene recognition model. The scene recognition model can identify and classify the input video frames to obtain the target scene type of the first video.

[0161] Step 7200: In the case that the target scenario is an online class scenario, the second translated text information is synthesized into speech to obtain the third speech information and played.

[0162] Understandably, considering that online course videos, compared to film and television videos, require a higher level of language comprehension, to minimize user difficulties in understanding the language, if audio translation is needed, the translated audio should not only be displayed as subtitles but also synthesized and played back. Furthermore, the audio in film and television dramas often contains complex and rich emotional elements; synthesizing and playing back the audio would result in the loss of the original emotion, so the original audio should be preserved.

[0163] In this embodiment, in order to reduce the difficulty of language comprehension for users in online class scenarios, the second translated text information corresponding to the main voice information is synthesized into a third voice information for playback.

[0164] Based on steps 7100 to 7200 above, it can classify videos based on the current screen to provide users with different translation strategies in different video scenarios, thereby improving the user's translation experience.

[0165] The following is an example of a text display method, in which reference... Figure 8 The text display method includes the following steps:

[0166] Step 701: While playing the first video, obtain the video frames of the first video and the corresponding audio information of the video frames.

[0167] Step 702: Detect whether the audio information needs to be translated. If yes, proceed to step 703; otherwise, proceed to step 708.

[0168] Step 703: When it is detected that the audio information needs to be translated, the background speech recognition model is used to identify that the audio information includes background speech information, and the first pitch feature and the first source text information of the background speech information are obtained. The pitch feature matrix corresponding to the first pitch feature is obtained through the annotation embedding layer, and the text feature matrix corresponding to the first source text information is obtained through the first attention network. The pitch feature matrix and the text feature matrix are fused through the second attention network to obtain the fused feature matrix. The first translated text information of the first source text information is determined based on the fused feature matrix through the fully connected layer.

[0169] Step 704: Determine the second translated text information of the main speech information in the audio information.

[0170] Step 705: Display the first translated text information and the second translated text information on the video frame.

[0171] Step 706: Determine if the first video is an online class scenario. If yes, proceed to step 707; otherwise, proceed to step 708.

[0172] Step 707: In the case that the first video is an online class scenario, the second translated text information is synthesized into speech to obtain the third speech information and played, and then step 708 is executed.

[0173] Step 708: Detect whether the video frame contains English text. If yes, proceed to step 709; otherwise, proceed to step 710.

[0174] Step 709: Based on the associated entity information and mapping data of the English text in the video frame, determine the third translation text information corresponding to the English text and display it near the English text.

[0175] Step 710: Determine if the first video has finished playing. If yes, the process ends; otherwise, proceed to step 701 above.

[0176] Corresponding to the above embodiments, see [link to relevant documentation]. Figure 9 This application embodiment also provides a text display device 900, which includes a first acquisition module 910, a second acquisition module 920, a third acquisition module 930, a first determination module 940, a second determination module 950, and a display module 960.

[0177] The first acquisition module 910 is used to acquire the audio information of the first video while the first video is playing.

[0178] The second acquisition module 920 is used to acquire the first pitch feature and the first source text information of the background speech information in the audio information;

[0179] The third acquisition module 930 is used to acquire the tone feature matrix corresponding to the first tone feature through the annotation embedding layer, and to acquire the text feature matrix corresponding to the first source text information through the first attention network.

[0180] The first determining module 940 is used to fuse the tone feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix, and to determine the first translated text information of the first source text information based on the fused feature matrix through a fully connected layer;

[0181] The second determining module 950 is used to determine the second translated text information of the main speech information in the audio information;

[0182] Display module 960 is used to display the first translated text information and the second translated text information.

[0183] In one embodiment, the first determining module 940 is specifically configured to: multiply the text feature matrix and the first parameter matrix of the fully connected layer using a second attention network to obtain a text feature transformation matrix; multiply the pitch feature matrix and the second parameter matrix of the fully connected layer to obtain a first pitch feature transformation matrix; multiply the pitch feature matrix and the third parameter matrix of the fully connected layer to obtain a second pitch feature transformation matrix; multiply the text feature transformation matrix and the first pitch feature transformation matrix to obtain an attention matrix; and multiply the attention matrix and the second pitch feature transformation matrix to obtain a fusion feature matrix.

[0184] In one embodiment, the device 900 further includes a first identification module, an extraction module, and a third determination module (all not shown in the figure).

[0185] The first recognition module is used to perform speech recognition on the audio information to obtain target source text information;

[0186] The extraction module is used to extract the target tone features of the target source text information;

[0187] The third determining module is used to determine whether the audio information includes background speech information based on the target tone features, the target source text information, and the preset background speech recognition model.

[0188] The second acquisition module 920 is further configured to acquire, when the audio information includes background speech information, a first pitch feature and a first source text information of the background speech information in the audio information.

[0189] In one embodiment, the device further includes a second identification module and a translation module (both not shown in the figure).

[0190] The first acquisition module 910 is further configured to acquire the video frame corresponding to the audio information;

[0191] The second recognition module is used to perform image text recognition on the video frame to obtain third source text information;

[0192] The translation module is used to translate the third source text information when the language of the third source text information is the target language, so as to obtain the third translated text information;

[0193] The display module 960 is also used to display the third translated text information.

[0194] In one embodiment, the translation module is specifically configured to: translate the third source text information to obtain at least one fourth translated text information corresponding to the third source text information; acquire set mapping data; wherein the mapping data reflects the correspondence between different fourth translated text information and entity information; and determine the third translated text information from the at least one fourth translated text information based on the associated entity information of the third source text information in the video frame and the mapping data.

[0195] In one embodiment, the device 900 further includes a third identification module and a fourth determination module (not shown in the figure).

[0196] The third recognition module is used to perform image text recognition on the first preset area of ​​the video frame when the language of the audio information is the target language, and obtain a first recognition result;

[0197] The fourth determining module is used to determine that the audio information needs to be translated when the first recognition result indicates that no text information has been recognized.

[0198] The fourth determining module is further configured to determine that the audio information needs to be translated when the first recognition result indicates that text information has been recognized and the language of the text information is the target language;

[0199] The second acquisition module 920 is further configured to acquire, when translation of the audio information is required, the first pitch feature of the background speech information and the first source text information in the audio information.

[0200] In the embodiments of this application, when playing a first video, the electronic device acquires the audio information of the first video, and automatically acquires the first pitch feature of the background speech information and the first source text information in the audio information. It then acquires the pitch feature matrix corresponding to the first pitch feature through an annotation embedding layer, and the text feature matrix corresponding to the first source text information through a first attention network. Finally, it fuses the pitch feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix. Based on the fused feature matrix, a fully connected layer determines the first translated text information of the first source text information. Simultaneously, it also determines the second translated text information of the main speech information in the audio information, and then displays the first and second translated text information. That is, when playing the first video, the electronic device can translate the audio information in the first video without switching to a translation application. Furthermore, it can distinguish between background speech information and main speech information, and display the first translated text information of the background speech information and the second translated text information of the main speech information. Moreover, it adaptively enhances the translation of the background speech information based on pitch features, thereby achieving high-quality translation and improving the user's viewing experience.

[0201] The text display device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0202] The text display device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0203] The text display device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0204] Optionally, such as Figure 10 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described text display method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0205] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0206] Figure 11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0207] The electronic device 2300 includes, but is not limited to, the following components: radio frequency unit 2301, network module 2302, audio output unit 2303, input unit 2304, sensor 2305, display unit 2306, user input unit 2307, interface unit 2308, memory 2309, processor 2310, etc.

[0208] Those skilled in the art will understand that the electronic device 2300 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 2310 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0209] The processor 2310 is configured to, while playing a first video, acquire audio information of the first video; acquire a first pitch feature and a first source text information of background speech information in the audio information; acquire a pitch feature matrix corresponding to the first pitch feature through an annotation embedding layer, and acquire a text feature matrix corresponding to the first source text information through a first attention network; fuse the pitch feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix, and determine the first translated text information of the first source text information based on the fused feature matrix through a fully connected layer; determine the second translated text information of the main speech information in the audio information; and display the first translated text information and the second translated text information.

[0210] According to an embodiment of this application, when playing a first video, the electronic device acquires the audio information of the first video, and automatically acquires the first pitch feature of the background speech information and the first source text information in the audio information. It then acquires the pitch feature matrix corresponding to the first pitch feature through an annotation embedding layer, and the text feature matrix corresponding to the first source text information through a first attention network. Finally, it fuses the pitch feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix. Based on the fused feature matrix, a fully connected layer determines the first translated text information of the first source text information. Simultaneously, it also determines the second translated text information of the main speech information in the audio information, and then displays the first and second translated text information. That is, when playing the first video, the electronic device can translate the audio information in the first video without switching to a translation application. Furthermore, it can distinguish between background speech information and main speech information, and display the first translated text information of the background speech information and the second translated text information of the main speech information. Moreover, it adaptively enhances the translation of the background speech information based on pitch features, thereby achieving high-quality translation and improving the user's viewing experience.

[0211] In one embodiment, the processor 2310 is configured to perform matrix multiplication of the text feature matrix and the first parameter matrix of the fully connected layer through a second attention network to obtain a text feature transformation matrix; perform matrix multiplication of the pitch feature matrix and the second parameter matrix of the fully connected layer to obtain a first pitch feature transformation matrix; perform matrix multiplication of the pitch feature matrix and the third parameter matrix of the fully connected layer to obtain a second pitch feature transformation matrix; perform matrix multiplication of the text feature transformation matrix and the first pitch feature transformation matrix to obtain an attention matrix; and perform matrix multiplication of the attention matrix and the second pitch feature transformation matrix to obtain a fused feature matrix.

[0212] In one embodiment, the processor 2310 is further configured to perform speech recognition on the audio information to obtain target source text information; extract target tone features from the target source text information; determine whether the audio information includes background speech information based on the target tone features, the target source text information, and a preset background speech recognition model; and if the audio information includes background speech information, obtain the first tone features and the first source text information of the background speech information in the audio information.

[0213] In one embodiment, the processor 2310 is configured to acquire a video frame corresponding to the audio information; perform image text recognition on the video frame to obtain third source text information; if the language of the third source text information is the target language, translate the third source text information to obtain third translated text information; and display the third translated text information.

[0214] In one embodiment, the processor 2310 is configured to translate the third source text information to obtain at least one fourth translated text information corresponding to the third source text information; acquire set mapping data; wherein the mapping data reflects the correspondence between different fourth translated text information and entity information; and determine the third translated text information from the at least one fourth translated text information based on the associated entity information of the third source text information in the video frame and the mapping data.

[0215] In one embodiment, the processor 2310 is configured to: perform image text recognition on a first preset region of the video frame to obtain a first recognition result when the language of the audio information is the target language; determine that the audio information needs to be translated when the first recognition result indicates that no text information has been recognized; determine that the audio information needs to be translated when the first recognition result indicates that text information has been recognized and the language of the text information is the target language; and obtain a first pitch feature of the background speech information and a first source text information in the audio information when the audio information needs to be translated.

[0216] It should be understood that, in this embodiment, the input unit 2304 may include a graphics processing unit (GPU) 23041 and a microphone 23042. The GPU 23041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 2306 may include a display panel 23061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 2307 includes at least one of a touch panel 23071 and other input devices 23072. The touch panel 23071 is also called a touch screen. The touch panel 23071 may include a touch detection device and a touch controller. Other input devices 23072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0217] The memory 2309 can be used to store software programs and various data. The memory 2309 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 2309 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 2309 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0218] Processor 2310 may include one or more processing units; optionally, processor 2310 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 2310.

[0219] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described text display method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0220] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0221] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described text display method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0222] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0223] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described text display method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0224] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0225] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0226] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A text display method, characterized in that, The method includes: When playing the first video, obtain the audio information of the first video; Obtain the first pitch feature and the first source text information of the background speech information in the audio information; The tone feature matrix corresponding to the first tone feature is obtained through the annotation embedding layer, and the text feature matrix corresponding to the first source text information is obtained through the first attention network. The pitch feature matrix and the text feature matrix are fused using a second attention network to obtain a fused feature matrix, and the first translated text information of the first source text information is determined based on the fused feature matrix using a fully connected layer. Determine the second translated text information of the main speech information in the audio information; Display the first translated text information and the second translated text information; The step after obtaining the audio information of the first video includes: The audio information is subjected to speech recognition to obtain the target source text information; Extract the target pitch features from the target source text information; Based on the target pitch features, the target source text information, and the preset background speech recognition model, it is determined whether the audio information includes background speech information; If the audio information includes background speech information, the first pitch feature and the first source text information of the background speech information in the audio information are obtained.

2. The method according to claim 1, characterized in that, The process of fusing the pitch feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix includes: The text feature matrix is ​​obtained by performing matrix multiplication between the text feature matrix and the first parameter matrix of the fully connected layer through the second attention network; The first pitch feature transformation matrix is ​​obtained by performing matrix multiplication between the pitch feature matrix and the second parameter matrix of the fully connected layer. The second pitch feature transformation matrix is ​​obtained by performing matrix multiplication between the pitch feature matrix and the third parameter matrix of the fully connected layer. The attention matrix is ​​obtained by performing matrix multiplication between the text feature transformation matrix and the first tone feature transformation matrix; The attention matrix and the second pitch feature transformation matrix are multiplied together to obtain the fused feature matrix.

3. The method according to claim 1, characterized in that, In the case of playing the first video, it also includes: Obtain the video frame corresponding to the audio information; Perform image text recognition on the video frames to obtain third-party source text information; If the language of the third source text information is the target language, the third source text information is translated to obtain the third translated text information; Display the third translated text information.

4. The method according to claim 3, characterized in that, The process of translating the third source text information to obtain third translated text information includes: The third source text information is translated to obtain at least one fourth translated text information corresponding to the third source text information; Obtain the set mapping data; wherein the mapping data reflects the correspondence between different fourth translation text information and entity information; The third translated text information is determined from the at least one fourth translated text information based on the associated entity information in the video frame and the mapping data of the third source text information.

5. The method according to claim 3, characterized in that, After obtaining the video frame corresponding to the audio information, the process further includes: If the language of the audio information is the target language, perform image text recognition on the first preset region of the video frame to obtain a first recognition result; If the first recognition result indicates that no text information is recognized, it is determined that the audio information needs to be translated; If the first recognition result indicates that text information has been recognized and the language of the text information is the target language, it is determined that the audio information needs to be translated. When it is necessary to translate the audio information, the first pitch feature of the background speech information and the first source text information in the audio information are obtained.

6. A text display device, characterized in that, The device includes: The first acquisition module is used to acquire the audio information of the first video while the first video is playing; The second acquisition module is used to acquire the first pitch feature and the first source text information of the background speech information in the audio information; The third acquisition module is used to acquire the pitch feature matrix corresponding to the first pitch feature through the annotation embedding layer, and to acquire the text feature matrix corresponding to the first source text information through the first attention network. The first determining module is used to fuse the pitch feature matrix and the text feature matrix through a second attention network to obtain a fused feature matrix, and to determine the first translated text information of the first source text information based on the fused feature matrix through a fully connected layer; The second determining module is used to determine the second translated text information of the main speech information in the audio information; The display module is used to display the first translated text information and the second translated text information; The device further includes a first identification module, an extraction module, and a third determination module. The first recognition module is used to perform speech recognition on the audio information to obtain target source text information; The extraction module is used to extract the target tone features of the target source text information; The third determining module is used to determine whether the audio information includes background speech information based on the target tone features, the target source text information, and the preset background speech recognition model. The second acquisition module is further configured to acquire, when the audio information includes background speech information, a first pitch feature and a first source text information of the background speech information in the audio information.

7. The apparatus according to claim 6, characterized in that, The first determining module is specifically used for: The text feature matrix is ​​obtained by performing matrix multiplication between the text feature matrix and the first parameter matrix of the fully connected layer through the second attention network; The first pitch feature transformation matrix is ​​obtained by performing matrix multiplication between the pitch feature matrix and the second parameter matrix of the fully connected layer. The second pitch feature transformation matrix is ​​obtained by performing matrix multiplication between the pitch feature matrix and the third parameter matrix of the fully connected layer. The attention matrix is ​​obtained by performing matrix multiplication between the text feature transformation matrix and the first tone feature transformation matrix; The attention matrix and the second pitch feature transformation matrix are multiplied together to obtain the fused feature matrix.

8. The apparatus according to claim 6, characterized in that, The device also includes a second identification module and a translation module. The first acquisition module is further configured to acquire the video frame corresponding to the audio information; The second recognition module is used to perform image text recognition on the video frame to obtain third source text information; The translation module is used to translate the third source text information when the language of the third source text information is the target language, so as to obtain the third translated text information; The display module is also used to display the third translated text information.

9. The apparatus according to claim 8, characterized in that, The translation module is specifically used for: The third source text information is translated to obtain at least one fourth translated text information corresponding to the third source text information; Obtain the set mapping data; wherein the mapping data reflects the correspondence between different fourth translation text information and entity information; The third translated text information is determined from the at least one fourth translated text information based on the associated entity information in the video frame and the mapping data of the third source text information.

10. The apparatus according to claim 8, characterized in that, The device also includes a third identification module and a fourth determination module. The third recognition module is used to perform image text recognition on the first preset area of ​​the video frame when the language of the audio information is the target language, and obtain a first recognition result; The fourth determining module is used to determine that the audio information needs to be translated when the first recognition result indicates that no text information has been recognized. The fourth determining module is further configured to determine that the audio information needs to be translated when the first recognition result indicates that text information has been recognized and the language of the text information is the target language; The second acquisition module is further configured to acquire, when it is necessary to translate the audio information, the first pitch feature of the background speech information and the first source text information in the audio information.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the text display method as described in any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the text display method as described in any one of claims 1-5.

Images