Subtitle processing method, device, equipment and storage medium

Abstract

Provided is a subtitle processing method, device, equipment and storage medium, wherein the method includes: detecting that a user enters a video conference scene, obtaining the main language information of the current speaker of the video conference as the first language information, and obtaining the main language information of the user as the second language information; when the first language information is different from the second language information, obtaining the voice data of the speaker, and generating corresponding subtitles according to the voice data; determining the display style of the subtitles according to the historical translation-related operations of the user; and pushing the subtitles to the video conference scene of the user in real time according to the display style of the subtitles. It can solve the problem of cross-language communication in the video conference scene and meet the user's need to understand the content of the meeting in the video conference scene.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a subtitle processing method, device, equipment and storage medium. Background Art

[0002] With the popularization of global collaboration scenarios, people are no longer limited to communicating with people who speak the same language for work needs. Cross-language communication has become a work choice for more groups. In a video conference scenario, if the user is not familiar with the language of the conference speaker, it is difficult to understand the content of the conference, which reduces the effectiveness of the video conference. Based on this, it is necessary to provide a technical solution to solve the problem of cross-language communication in video conference scenarios and meet the user's need to understand the content of the conference in video conference scenarios. Summary of the invention

[0003] The embodiments of this specification provide a subtitle processing method, apparatus, device and storage medium to solve the problem of cross-language communication in a video conferencing scenario and meet the user's need to understand the content of the conference in a video conferencing scenario.

[0004] In a first aspect, an embodiment of the present specification provides a subtitle processing method, comprising:

[0005] Detecting that a user enters a video conference scene, obtaining primary language information of a current speaker of the video conference as first language information, and obtaining primary language information of the user as second language information;

[0006] When the first language information is different from the second language information, acquiring voice data of the speaker, and generating corresponding subtitles according to the voice data;

[0007] Determining a display style of the subtitles according to the user's historical translation-related operations;

[0008] According to the display style of the subtitles, the subtitles are pushed to the video conference scene of the user in real time.

[0009] In a second aspect, an embodiment of the present specification provides a subtitle processing device, including:

[0010] An information acquisition unit is used to detect that a user enters a video conference scene, acquire the primary language information of the current speaker of the video conference as the first language information, and acquire the primary language information of the user as the second language information;

[0011] a subtitle generating unit, configured to obtain voice data of the speaker and generate corresponding subtitles according to the voice data when the first language information is different from the second language information;

[0012] A style determination unit, configured to determine a display style of the subtitles according to the historical translation-related operations of the user;

[0013] A subtitle display unit is used to push the subtitles to the user's video conference scene in real time according to the display style of the subtitles.

[0014] In a third aspect, an embodiment of the present specification provides a subtitle processing device, including: a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the steps of the method described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in the first aspect above.

[0016] In one embodiment of the present specification, it is detected that a user enters a video conference scene, the main language information of the current speaker of the video conference is obtained as the first language information, and the main language information of the user is obtained as the second language information. When the first language information and the second language information are different, the voice data of the speaker is obtained, and corresponding subtitles are generated according to the voice data. According to the user's historical translation-related operations, the display style of the subtitles is determined, and according to the display style of the subtitles, the subtitles are pushed to the user's video conference scene in real time, so as to display the subtitles in real time in the user's video conference scene. It can be seen that through this embodiment, subtitles can be generated in real time according to the speaker's speech content in the video conference scene, and the display style of the subtitles can be determined according to the user's situation, and based on the display style of the subtitles, the subtitles can be displayed in real time in the user's video conference scene, thereby solving the cross-language communication problem in the video conference scene and meeting the user's need to understand the content of the meeting in the video conference scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate one or more embodiments of the present specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0018] Figure 1 A flowchart of a subtitle processing method provided in an embodiment of this specification;

[0019] Figure 2 A schematic diagram of a subtitle display style provided in an embodiment of this specification;

[0020] Figure 3a A schematic diagram of a scenario of turning on subtitles in a video conference provided in an embodiment of this specification;

[0021] Figure 3b A schematic diagram of a scenario of turning on subtitles in a video conference provided by another embodiment of the present specification;

[0022] Figure 4 A schematic diagram of the structure of a subtitle processing device provided in an embodiment of this specification;

[0023] Figure 5 A schematic diagram of the structure of a subtitle processing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0025] The embodiments of this specification provide a subtitle processing method, apparatus, device and storage medium to solve the problem of cross-language communication in a video conferencing scenario and meet the user's need to understand the content of the conference in a video conferencing scenario.

[0026] It is understandable that before using the technical solutions disclosed in the embodiments of this specification, the types, scope of use, usage scenarios, etc. of personal information involved in the embodiments of this specification should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0027] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution in the embodiment of this description according to the prompt message.

[0028] As an optional but non-limiting implementation, the prompt information may be sent to the user in a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present embodiment. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present embodiment.

[0030] Figure 1 A flowchart of a subtitle processing method provided in an embodiment of this specification is shown in FIG. Figure 1 As shown, the process includes the following steps:

[0031] Step S102, detecting that a user enters a video conference scene, obtaining the primary language information of the current speaker of the video conference as the first language information, and obtaining the primary language information of the user as the second language information;

[0032] Step S104, when the first language information is different from the second language information, acquiring the speaker's voice data, and generating corresponding subtitles according to the voice data;

[0033] Step S106, determining the display style of the subtitles according to the user's historical translation-related operations;

[0034] Step S108: Push the subtitles to the video conference scene of the user in real time according to the display style of the subtitles.

[0035] In one embodiment of the present specification, it is detected that a user enters a video conference scene, the main language information of the current speaker of the video conference is obtained as the first language information, and the main language information of the user is obtained as the second language information. When the first language information and the second language information are different, the voice data of the speaker is obtained, and corresponding subtitles are generated according to the voice data. According to the user's historical translation-related operations, the display style of the subtitles is determined, and according to the display style of the subtitles, the subtitles are pushed to the user's video conference scene in real time, so as to display the subtitles in real time in the user's video conference scene. It can be seen that through this embodiment, subtitles can be generated in real time according to the speaker's speech content in the video conference scene, and the display style of the subtitles can be determined according to the user's situation, and based on the display style of the subtitles, the subtitles can be displayed in real time in the user's video conference scene, thereby solving the cross-language communication problem in the video conference scene and meeting the user's need to understand the content of the meeting in the video conference scene.

[0036] The subtitle processing method in this embodiment can be applied to the server side and executed by the server, which can be a server of collaborative office software integrated with video conferencing applications. Alternatively, the subtitle processing method in this embodiment can be applied to the user terminal of the user and executed by the user terminal, which can be a smart terminal such as a mobile phone, computer, tablet computer, laptop computer, car computer, wearable device, etc. The user terminal can run collaborative office software integrated with video conferencing applications. Among them, collaborative office software is a software that integrates multiple office applications such as instant messaging, cloud documents, audio and video conferencing, etc., which can greatly improve the efficiency of collaborative office work between employees.

[0037] In the above step S102, it is detected whether the user has entered the video conference scene. If it is detected that the user has clicked the button to join the video conference, it is determined that the user has entered the video conference scene. After detecting that the user has entered the video conference scene, the main language information of the current speaker of the video conference is obtained as the first language information, and the main language information of the user is obtained as the second language information.

[0038] The current speaker of the video conference is the person who is speaking in the video conference at the current moment. The speaker of the video conference may change over time. In one embodiment, the main language information of the current speaker of the video conference is obtained, specifically:

[0039] (a1) obtaining speech data emitted by the speaker;

[0040] (a2) when the voice data corresponds to a language information, determining the corresponding language information as the main language information of the speaker;

[0041] (a3) When the speech data corresponds to multiple language information, the amount of audio data corresponding to each language information in the speech data is determined, and the language information corresponding to the largest amount of audio data is determined as the main language information of the speaker.

[0042] In a video conference scenario, all participants can communicate through voice. Based on this, the voice data sent by the current speaker in the video conference can be obtained, and the voice data can be determined to correspond to several language information. Specifically, it is determined how many languages ​​are included in the voice data, so as to determine how many language information the voice data corresponds to. For example, the voice data "hello" includes one language - Chinese, and thus corresponds to one language information - Chinese; the voice data "hello, have you eaten?" includes two languages ​​- Chinese and English, and thus corresponds to two language information - Chinese and English.

[0043] Next, if it is determined that the speaker's voice data corresponds to one language information, the corresponding language information is determined as the speaker's main language information. If it is determined that the speaker's voice data corresponds to multiple language information, the amount of audio data corresponding to each language information in the speaker's voice data is determined, and the language information with the largest amount of corresponding audio data is determined as the speaker's main language information.

[0044] For example, if the speaker's voice data is "Hello, have you eaten? How's the weather today?", the amount of audio data corresponding to the various language information (Chinese and English) in the voice data can be analyzed, and the language information (Chinese) with the largest amount of corresponding audio data can be determined as the speaker's main language information.

[0045] It can be seen that through this embodiment, when the speaker's voice data corresponds to multiple language information, the speaker's main language information can be determined according to the amount of audio data corresponding to the various language information in the speaker's voice data, thereby accurately determining the speaker's main language information in the video conferencing scenario.

[0046] In one example, the main language information of the speaker can be determined periodically, for example, the main language information of the speaker can be determined every seven days according to the voice messages and / or text messages sent by the speaker in various applications integrated with the collaborative office software, such as instant messaging applications and video conferencing applications, within these seven days. In another example, the language of the conversation messages that the speaker has sent in various applications integrated with the collaborative office software, such as instant messaging applications and video conferencing applications, can be identified by a pre-trained machine learning model. For example, the conversation messages sent by the speaker in the instant messaging application and the video conferencing application are input into the pre-trained machine learning model, and the machine learning model outputs the language information of the conversation message. The language information output by the machine learning model is used as the language label of the conversation message. In this way, the language labels of the conversation messages sent by the speaker in various applications integrated with the collaborative office software within the recent period (such as 7 days) can be counted to determine the main language information of the speaker. Among them, the conversation message can include voice messages and text messages. Of course, if the speaker sets a label of the main language information for himself, the label can be directly obtained to determine the main language information of the speaker.

[0047] In the above step S102, after determining the main language information of the speaker, the main language information of the speaker is determined as the first language information. In the above step S102, the main language information of the user is also obtained as the second language information. In one embodiment, the main language information of the user is obtained, specifically: using a machine learning model to identify the language of the conversation messages that the user has sent in various applications integrated with the collaborative office software, such as instant messaging applications and video conferencing applications. For example, the conversation messages sent by the user in the instant messaging application and the video conferencing application are input into a pre-trained machine learning model, and the machine learning model outputs the language information of the conversation message. The language information output by the machine learning model is used as the language label of the conversation message. In this way, the language labels of the conversation messages sent by the user in various applications integrated with the collaborative office software within a recent period of time (such as 3 months) can be counted to determine the main language information of the user. Of course, if the user sets a label for the main language information for himself, the label can be directly obtained to determine the main language information of the user. In this embodiment, after obtaining the main language information of the user, the main language information of the user is used as the second language information.

[0048] In the above step S104, it is determined whether the first language information and the second language information are the same. If they are the same, the subtitles in the video conference can be kept in a disabled state. If they are not the same, the voice data of the speaker speaking during the video conference is obtained, and corresponding subtitles are generated according to the voice data. The voice data can be converted into corresponding subtitles through voice recognition technology.

[0049] In the above step S106, the display style of the subtitles is determined according to the user's historical translation-related operations. The display style of the subtitles includes displaying the translation of part of the subtitles, displaying the translation of all the subtitles, and not displaying the translation of the subtitles.

[0050] Figure 2 A schematic diagram of the display style of subtitles provided in an embodiment of this specification is shown as follows: Figure 2 As shown, taking the same English subtitle as an example, this embodiment provides three display styles, from top to bottom: displaying the translation of part of the text in the subtitle, displaying the translation of the entire text in the subtitle, and not displaying the translation of the subtitle.

[0051] For users with strong cross-language ability, they may only need subtitles for the speaker of the video conference, so the display style of the subtitles can be determined to not display the translation of the subtitles. For users with certain cross-language ability, they may need to display the translation of uncommon words said by the speaker of the video conference, so the display style of the subtitles can be determined to display the translation of part of the text in the subtitles. For users with weak cross-language ability, they may need the subtitles of the speaker of the video conference and the corresponding full translation, so the display style of the subtitles can be determined to display the translation of all the text in the subtitles.

[0052] In one embodiment, the display style of the subtitles is determined according to the historical translation-related operations of the user, including at least one of the following:

[0053] (b1) if it is determined based on the user's historical translation-related operations that the user has enabled the subtitle translation function for the video conference, then determining that the display style of the subtitles is to display the translation of all text in the subtitles;

[0054] (b2) if it is determined based on the user's historical translation-related operations that the user has not enabled the subtitle translation function for the video conference, and the frequency of the user's historical execution of the specified translation operation is greater than or equal to a preset frequency, then the subtitle display style is determined to be a translation of a portion of the text in the subtitle;

[0055] (b3) if it is determined based on the user's historical translation-related operations that the user has not enabled the subtitle translation function for the video conference, and the frequency of the user's historical execution of the specified translation operation is less than a preset frequency, then the subtitle display style is determined to be a translation without displaying the subtitles;

[0056] The designated translation operation is a translation operation for the first language information.

[0057] The user's historical translation-related operations include turning on and off the subtitle translation function for the video conference. First, based on the user's historical translation-related operations, it is determined whether the user has turned on the subtitle translation function in the video conference. If so, it is determined that the user has turned on the subtitle translation function for the video conference, thereby determining that the user needs all translations to understand the content of the video conference, and determining that the display style of the subtitles is to display the translation of all texts in the subtitles.

[0058] If it is determined that the user has not turned on the subtitle translation function in the video conference, that is, it is determined that the user has not turned on the subtitle translation function for the video conference, then based on the user's historical translation-related operations, it is determined whether the frequency of the user's historical execution of the designated translation operation is greater than or equal to the preset frequency. The designated translation operation is the user's translation operation for the first language information.

[0059] In one embodiment, the video conferencing application in which the video conferencing scenario is located is integrated into collaborative office software, and the user's historical translation-related operations include the translation operations historically performed by the user in various applications integrated with the collaborative office software. It can be determined whether the frequency of the user's historical execution of the specified translation operation is greater than or equal to the preset frequency based on the translation operations historically performed by the user in various applications integrated with the collaborative office software.

[0060] Specifically, the various translation operations historically performed by the user in various applications (such as video conferencing applications, instant messaging applications, translation applications, etc.) integrated in the collaborative office software that provides the video conferencing function can be obtained, and the designated translation operation for translating the language corresponding to the first language information can be extracted from the obtained translation operations. For example, the various translation operations performed by the user in various applications integrated in the collaborative office software within one week before the current time are obtained, and the designated translation operation for translating the language corresponding to the first language information is extracted from the obtained translation operations. The designated translation operation for translating the language corresponding to the first language information can be the translation operation used by the user to translate the language corresponding to the first language information under any circumstances. For example, the translation operation for the first language information in a single chat, group chat or video conference scenario with the above-mentioned video conference speaker, or the translation operation for the first language information in a single chat, group chat or video conference scenario with other users who use the first language, or the translation operation for the first language information in the translation application integrated by the collaborative office software.

[0061] Determine whether the frequency of the user's historical execution of the specified translation operation is greater than or equal to a preset frequency, for example, whether it is greater than or equal to 2 times / week. If so, it is determined that the user's ability to read the first language information is weak, and it is determined that the user needs a partial translation of the above subtitles, so the display style of the subtitles is determined to display the translation of a portion of the text in the subtitles.

[0062] On the contrary, if it is determined that the user has not turned on the subtitle translation function for the video conference, and the frequency of the user's historical execution of the specified translation operation is less than the preset frequency, it means that the user has a strong ability to read information in the first language, and the display style of the subtitles is determined to be not displaying the translation of the subtitles.

[0063] In one embodiment, in a video conference, if the user turns off the subtitles after the subtitles are turned on for the user, the subtitles will no longer be automatically turned off in subsequent video conferences. Figure 1 The process in turns on subtitles for users.

[0064] It can be seen that through this embodiment, the display style of the subtitles corresponding to the user can be determined in a targeted manner according to the user's historical translation-related operations, so that the display style of the subtitles matches the user's language ability, bringing a better video conferencing experience to the user.

[0065] In one embodiment, after determining that the display style of the subtitle is to display the translation of part of the text in the subtitle, the part of the content in the subtitle that needs to display the translation is determined through the following process:

[0066] (d1) obtaining the user's target vocabulary;

[0067] (d2) comparing the subtitles with words in the target vocabulary to determine the first target word in the subtitles that is located in the target vocabulary;

[0068] (d3) Determine the translation of the first target word in the displayed subtitles.

[0069] First, the user's target word library is obtained. The target word library records the uncommon words for the user in the first language information. Then, the subtitles are compared with the words in the target word library one by one to determine the first target word in the target word library in the subtitles. It can be understood that the first target word is the uncommon word in the subtitles for the user in the first language information. Finally, the translation of the first target word in the subtitles is determined and displayed. Of course, the translation belongs to the second language information corresponding to the user.

[0070] It can be seen that by comparing the subtitles with the user's corresponding uncommon word library, it is possible to accurately determine the uncommon words for the user in the first language information in the subtitles, and determine to display the translation of these uncommon words in the second language information, thereby facilitating the user to understand the content of the video conference and improving the user's video conference communication efficiency.

[0071] In one embodiment, the user's target vocabulary is obtained as follows:

[0072] (d11) obtaining a pre-established first vocabulary, and using the first vocabulary as a target vocabulary of the user;

[0073] or,

[0074] (d12) determining a vocabulary that matches the user's language features from among a plurality of pre-established second vocabulary libraries according to the user's historical translation-related operations, and using the matching vocabulary library as the user's target vocabulary library;

[0075] or,

[0076] (d13) According to the user's historical translation related operations, determine the words that the user has translated historically, obtain a pre-established third vocabulary, add the words that the user has translated historically to the third vocabulary, and obtain the user's target vocabulary.

[0077] In one case, a first vocabulary is pre-established. The first vocabulary is a rare vocabulary that is common to any user. The first vocabulary can be, for example, a vocabulary containing first-level (inclusive) and above words in the English field. The first vocabulary is used as the user's target vocabulary.

[0078] In another case, the level of the words historically translated by the user is determined based on the translation operations historically performed by the user in various applications integrated by the collaborative office software, and then a word library matching the level of the words historically translated by the user is selected from the multiple pre-established second word libraries as the word library matching the user's language features, and then as the user's target word library. For example, if the words historically translated by the user belong to the first level and above in the English field, then a word library including words of the first level and above in the English field is selected from the multiple pre-established second word libraries as the user's target word library.

[0079] In another case, the words historically translated by the user are determined according to the historical translation-related operations of the user, including the translation operations historically performed by the user in various applications integrated with the collaborative office software. A pre-established third word library is obtained, which is also a rare word library common to any user, and the third word library can be the same as the first word library. The words historically translated by the user are added to the third word library to obtain the user's target word library.

[0080] It can be seen that in this embodiment, in one case, according to the level of the words translated by the user in the past, a word library matching the level of the words translated by the user in the past can be selected as the user's target word library, and in another case, the words translated by the user in the past can be added to the general word library to form a user-personalized uncommon word library. Thus, the user-personalized uncommon word library is obtained in a variety of ways, so that when determining the translation of part of the text in the subtitles, the words that the user is not familiar with can be accurately determined and translated.

[0081] In one embodiment, under the above process, the following may also be performed: determining in the user's target word library a second target word whose number of translations in the user's video conference scenario reaches a preset number threshold, and marking the second target word as a word whose translation is suspended in the user's video conference scenario. Since the second target word has been translated many times, it is inferred that the user has mastered the word and its translation, and therefore the second target word is marked as a word whose translation is suspended in the user's video conference scenario, so as not to provide the user with the translation of the second target word in the video conference scenario, thereby saving system workload.

[0082] In the above step S108, after determining the display style of the subtitles, the subtitles are pushed to the user's video conference scene in real time according to the display style of the subtitles, so that the subtitles are displayed to the user in real time in the video conference scene to assist the user in understanding the content of the conference. It should be noted that pushing the subtitles to the user's video conference scene in real time means pushing the subtitles to the user's video conference interface so that the subtitles are displayed in the user's video conference interface.

[0083] Figure 3a A schematic diagram of a scenario of opening subtitles in a video conference provided in an embodiment of this specification, such as Figure 3a As shown, you can automatically turn on subtitles for users in a video conference and customize the display style of subtitles. Figure 3a The display style of showing all translations is used as an example for illustration.

[0084] Figure 3b A schematic diagram of a scene of opening subtitles in a video conference provided by another embodiment of the present specification, and Figure 3a correspond, Figure 3b In the video, after the subtitles are turned on, the full translation of the subtitles can be displayed to the user in real time.

[0085] It can be seen that through the above embodiments, the subtitles of the video conference can be automatically turned on for the user in the video conference scenario, thereby improving the user's video experience. It should be noted that before executing this embodiment, the user may be prompted whether to agree to authorize the automatic turning on of the subtitle recommendation function for the video conference, and the user may be explained that when the subtitle recommendation function for the video conference is automatically turned on, the user's necessary personal information needs to be obtained to help the user analyze the display style of the subtitles when the subtitles are turned on. If the user clicks to agree to automatically turn on the subtitle recommendation function for the video conference, the above process can be used to automatically turn on the subtitles for the user in the video conference scenario and determine the display style of the subtitles without infringing the user's privacy.

[0086] In one embodiment, if it is determined that the display style of the subtitles is to display the translation of part of the text in the subtitles, then accordingly, according to the display style of the subtitles, the subtitles are pushed to the user's video conferencing scene in real time, which can be: displaying the subtitles and the translation of part of the text in the subtitles in the subtitle display area in the user's video conferencing scene; wherein, if the same word to be translated appears at least twice in the subtitles currently displayed in the subtitle display area, then the translation of the word to be translated that is located at the front of the subtitles is displayed.

[0087] Specifically, when displaying subtitles and translations of part of the text in the subtitles to the user, if a word to be translated appears multiple times in the subtitle display area, the word that is located at the front of the subtitles in each occurrence is determined in the subtitles currently displayed, and the translation of the word located at the front is displayed. For subsequent occurrences of the word in the currently displayed subtitles, the translation is not displayed, thereby saving space in the subtitle display area. The word located at the front may be the word that is spoken first in the order of the speaker's speech, and the word is located at the front in the order of the text from left to right in the currently displayed subtitles.

[0088] For example, in the subtitle display area in a video conference scene, the subtitles currently displayed are "apple, pear, apple", where "apple" is the word that needs to be translated. If the word appears twice, only the translation of the first appearance of the word is displayed, and the translation of the second appearance of the word is not displayed. Accordingly, the subtitles and their translations are "apple, pear, apple".

[0089] It should be noted that the subtitles displayed in the subtitle display area in the video conferencing scene will change as the speaker progresses. If according to the above process, it is determined that a word that needs to be translated appears twice in the subtitles currently displayed in the subtitle display area, and the translation of the word at the front position in the subtitle is displayed, if the subtitles in the subtitle display area change and the word appears again in the changed subtitles, the above process will be re-displayed to facilitate the user to understand the meaning of the word in the switched subtitles.

[0090] In one embodiment, in response to the user's subtitle activation operation, the video conference subtitles can be displayed in the user's video conference scene, and in response to the user's translation operation for the third target word in the conference subtitles, the translation of the third target word can be displayed in real time in the video conference scene.

[0091] The conference subtitles are the subtitles generated based on the speaker's voice data in the previous text. The subtitles can be generated based on the speaker's voice data through voice recognition technology according to the previous process. Specifically, Figure 1 The process in is a process of automatically turning on subtitles for users and determining the subtitle display style. In other embodiments, users can also manually turn on subtitles. After the user manually turns on subtitles, the subtitles do not display translated text by default. Users can translate the third target word in the subtitles, and the translated text of the third target word is displayed in the video conference scene, thereby providing the user with the required translation function according to the user's operation in the video conference scene.

[0092] In summary, through this embodiment, subtitles can be generated in real time according to the speaker's speech content in a video conferencing scenario, and the display style of the subtitles can be determined according to the user's situation. Based on the display style of the subtitles, the subtitles are displayed in real time in the user's video conferencing scenario, thereby solving the problem of cross-language communication in the video conferencing scenario and meeting the user's needs for understanding the content of the meeting in the video conferencing scenario.

[0093] Figure 4 A schematic diagram of the structure of a subtitle processing device provided in an embodiment of this specification is shown in FIG. Figure 4 As shown, the device comprises:

[0094] An information acquisition unit 41 is used to detect that a user enters a video conference scene, acquire the primary language information of the current speaker of the video conference as the first language information, and acquire the primary language information of the user as the second language information;

[0095] a subtitle generating unit 42, configured to obtain the speech data of the speaker and generate corresponding subtitles according to the speech data when the first language information is different from the second language information;

[0096] A style determination unit 43, configured to determine a display style of the subtitles according to the user's historical translation-related operations;

[0097] The subtitle display unit 44 is used to push the subtitles to the user's video conference scene in real time according to the display style of the subtitles.

[0098] Optionally, the display style of the subtitles includes displaying the translation of part of the text in the subtitles, displaying the translation of all the text in the subtitles, and not displaying the translation of the subtitles.

[0099] Optionally, the information acquisition unit 41 is specifically configured to:

[0100] Acquiring voice data of the speaker;

[0101] When the voice data corresponds to a language information, determining the corresponding language information as the main language information of the speaker;

[0102] When the voice data corresponds to multiple language information, the amount of audio data corresponding to each of the language information in the voice data is determined, and the language information corresponding to the largest amount of audio data is determined as the main language information of the speaker.

[0103] Optionally, the style determination unit 43 specifically includes at least one of the following:

[0104] If it is determined that the user has enabled the subtitle translation function for the video conference based on the historical translation-related operations of the user, then the display style of the subtitles is determined to be to display the translation of all the text in the subtitles;

[0105] If it is determined according to the historical translation-related operations of the user that the user has not enabled the subtitle translation function for the video conference, and the frequency of the user's historical execution of the specified translation operation is greater than or equal to a preset frequency, then determining that the display style of the subtitles is to display the translation of part of the text in the subtitles;

[0106] If it is determined based on the historical translation-related operations of the user that the user has not enabled the subtitle translation function for the video conference, and the frequency of the user's historical execution of the specified translation operation is less than the preset frequency, then the display style of the subtitles is determined to be not displaying the translation of the subtitles;

[0107] The designated translation operation is a translation operation for the first language information.

[0108] Optionally, the video conferencing application where the video conferencing scenario is located is integrated into collaborative office software; the device also includes: a judgment unit, used to judge whether the frequency of the user's historical execution of the specified translation operation is greater than or equal to the preset frequency based on the translation operations historically performed by the user in various applications integrated in the collaborative office software.

[0109] Optionally, a vocabulary comparison unit is also included, which is used to:

[0110] After determining that the display style of the subtitle is to display the translation of a portion of the text in the subtitle, obtaining the target vocabulary of the user;

[0111] Comparing the subtitles with words in the target vocabulary to determine, in the subtitles, a first target word in the target vocabulary;

[0112] Determine to display a translation of the first target word in the subtitle.

[0113] Optionally, the vocabulary comparison unit is specifically used for:

[0114] Acquire a pre-established first word library, and use the first word library as a target word library for the user;

[0115] or,

[0116] According to the historical translation-related operations of the user, determining a word library that matches the language features of the user from a plurality of pre-established second word libraries, and using the matching word library as the target word library of the user;

[0117] or,

[0118] According to the historical translation related operations of the user, the words historically translated by the user are determined, and the words historically translated by the user are added to a pre-established third word library to obtain the target word library of the user.

[0119] Optionally, a marking unit is further included, for:

[0120] Determine in the target word library a second target word whose number of translations in the video conference scenario of the user reaches a preset number threshold;

[0121] The second target word is marked as a word for which translation is suspended in the video conference scene of the user.

[0122] Optionally, if it is determined that the display style of the subtitle is to display the translation of part of the text in the subtitle, the subtitle display unit 44 is specifically used to:

[0123] Displaying the subtitles and a translation of a portion of the text in the subtitles in a subtitle display area in the user's video conference scene;

[0124] If the same word to be translated appears at least twice in the subtitles currently displayed in the subtitle display area, the translation of the word to be translated that is located at the front of the subtitles is displayed.

[0125] Optionally, a manual translation unit is also included for:

[0126] In response to a subtitle-enabling operation of the user, displaying conference subtitles of the video conference in the video conference scene of the user;

[0127] In response to the user's translation operation on the third target word in the conference subtitles, the translation of the third target word is displayed in real time in the video conference scene.

[0128] It should be noted that the subtitle processing device in this embodiment can implement each process of the aforementioned subtitle processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0129] An embodiment of the present specification also provides a subtitle processing device, Figure 5 A schematic diagram of the structure of a subtitle processing device provided in an embodiment of this specification is shown in FIG. Figure 5As shown, the subtitle processing device may have relatively large differences due to different configurations or performances, and may include one or more processors 801 and memory 802, and the memory 802 may store one or more applications or data. Among them, the memory 802 may be a temporary storage or a permanent storage. The application stored in the memory 802 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the subtitle processing device. Furthermore, the processor 801 may be configured to communicate with the memory 802, and execute a series of computer executable instructions in the memory 802 on the subtitle processing device. The subtitle processing device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input or output interfaces 805, one or more keyboards 806, etc.

[0130] Through the wired or wireless network interface 804, the subtitle processing device can be connected to a network, such as the Internet or a local area network, so as to communicate with other devices in the network. Through the input or output interface 805, the subtitle processing device can receive data input by an external device, or output data to an external device. In one example, the input or output interface 805 includes but is not limited to a touch display screen. Through the keyboard 806, the subtitle processing device can receive information input by the user and perform corresponding processing.

[0131] In a specific embodiment, the subtitle processing device may be a server or a user terminal, comprising a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0132] Detecting that a user enters a video conference scene, obtaining primary language information of a current speaker of the video conference as first language information, and obtaining primary language information of the user as second language information;

[0133] When the first language information is different from the second language information, acquiring voice data of the speaker, and generating corresponding subtitles according to the voice data;

[0134] Determining a display style of the subtitles according to the user's historical translation-related operations;

[0135] According to the display style of the subtitles, the subtitles are pushed to the video conference scene of the user in real time.

[0136] It should be noted that the subtitle processing device in this embodiment can implement each process of the aforementioned subtitle processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0137] Another embodiment of the present specification further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0138] Detecting that a user enters a video conference scene, obtaining primary language information of a current speaker of the video conference as first language information, and obtaining primary language information of the user as second language information;

[0139] When the first language information is different from the second language information, acquiring voice data of the speaker, and generating corresponding subtitles according to the voice data;

[0140] Determining a display style of the subtitles according to the user's historical translation-related operations;

[0141] According to the display style of the subtitles, the subtitles are pushed to the video conference scene of the user in real time.

[0142] It should be noted that the storage medium in this embodiment can implement each process of the aforementioned subtitle processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0143] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0144] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0145] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0146] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0147] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0148] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0153] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A subtitle processing method, characterized in that: include: Detecting that a user enters a video conference scene, obtaining primary language information of a current speaker of the video conference as first language information, and obtaining primary language information of the user as second language information; When the first language information is different from the second language information, acquiring voice data of the speaker, and generating corresponding subtitles according to the voice data; Determining a display style of the subtitles according to the historical translation-related operations of the user; the display style of the subtitles includes any one of the following: displaying a translation of a portion of the text in the subtitles, displaying a translation of all the text in the subtitles, or not displaying a translation of the subtitles; The determining of the display style of the subtitles according to the historical translation-related operations of the user specifically includes: determining, according to the historical translation-related operations of the user, whether the user has enabled the subtitle translation function for the video conference and whether the frequency of the user's historical execution of the designated translation operation is greater than or equal to a preset frequency, thereby determining the display style of the subtitles; the designated translation operation is a translation operation for the first language information; According to the display style of the subtitles, the subtitles are pushed to the video conference scene of the user in real time.

2. The method according to claim 1, characterized in that The obtaining of the subject language information of the current speaker of the video conference includes: Acquiring voice data of the speaker; When the voice data corresponds to a language information, determining the corresponding language information as the main language information of the speaker; When the voice data corresponds to multiple language information, the amount of audio data corresponding to each of the language information in the voice data is determined, and the language information corresponding to the largest amount of audio data is determined as the main language information of the speaker.

3. The method according to claim 1, characterized in that The determining of the display style of the subtitles according to the historical translation-related operations of the user includes at least one of the following: If it is determined that the user has enabled the subtitle translation function for the video conference based on the historical translation-related operations of the user, then the display style of the subtitles is determined to be to display the translation of all the text in the subtitles; If it is determined according to the historical translation-related operations of the user that the user has not enabled the subtitle translation function for the video conference, and the frequency of the user's historical execution of the specified translation operation is greater than or equal to the preset frequency, then determining that the display style of the subtitles is to display the translation of part of the text in the subtitles; If it is determined based on the user's historical translation-related operations that the user has not enabled the subtitle translation function for the video conference, and the user's historical frequency of performing the specified translation operation is less than the preset frequency, then the display style of the subtitles is determined to be not displaying the translation of the subtitles.

4. The method according to claim 3, characterized in that The video conferencing application in which the video conferencing scenario is located is integrated into collaborative office software; the method further includes: According to the translation operations historically performed by the user in various applications integrated with the collaborative office software, it is determined whether the frequency of the user's historical execution of the designated translation operation is greater than or equal to the preset frequency.

5. The method according to claim 1, characterized in that If it is determined that the display style of the subtitle is to display the translation of part of the text in the subtitle, the method further includes: Acquire a target vocabulary of the user; Comparing the subtitles with words in the target vocabulary to determine a first target word in the target vocabulary in the subtitles; Determine to display a translation of the first target word in the subtitle.

6. The method according to claim 5, characterized in that The step of obtaining the target vocabulary of the user includes: Acquire a pre-established first word library, and use the first word library as a target word library for the user; or, According to the historical translation-related operations of the user, determining a word library that matches the language features of the user from a plurality of pre-established second word libraries, and using the matching word library as the target word library of the user; or, According to the historical translation related operations of the user, the words historically translated by the user are determined, and the words historically translated by the user are added to a pre-established third word library to obtain the target word library of the user.

7. The method according to claim 5, characterized in that The method further comprises: Determine in the target word library a second target word whose number of translations in the video conference scenario of the user reaches a preset number threshold; The second target word is marked as a word for which translation is suspended in the video conference scene of the user.

8. The method according to claim 1, characterized in that If it is determined that the display style of the subtitles is to display the translation of part of the text in the subtitles, pushing the subtitles to the video conference scene of the user in real time according to the display style of the subtitles includes: Displaying the subtitles and a translation of a portion of the text in the subtitles in a subtitle display area in the user's video conference scene; If the same word to be translated appears at least twice in the subtitles currently displayed in the subtitle display area, the translation of the word to be translated that is located at the front of the subtitles is displayed.

9. The method according to claim 1, characterized in that: The method further comprises: In response to a subtitle-enabling operation of the user, displaying conference subtitles of the video conference in the video conference scene of the user; In response to the user's translation operation on the third target word in the conference subtitles, the translation of the third target word is displayed in real time in the video conference scene.

10. A subtitle processing device, characterized in that: include: An information acquisition unit is used to detect that a user enters a video conference scene, acquire the primary language information of the current speaker of the video conference as the first language information, and acquire the primary language information of the user as the second language information; a subtitle generating unit, configured to obtain voice data of the speaker and generate corresponding subtitles according to the voice data when the first language information is different from the second language information; A style determination unit, configured to determine a display style of the subtitles according to the historical translation-related operations of the user; the display style of the subtitles includes any one of the following: displaying a translation of a portion of the text in the subtitles, displaying a translation of all the text in the subtitles, or not displaying a translation of the subtitles; The determining of the display style of the subtitles according to the historical translation-related operations of the user specifically includes: determining, according to the historical translation-related operations of the user, whether the user has enabled the subtitle translation function for the video conference and whether the frequency of the user's historical execution of the designated translation operation is greater than or equal to a preset frequency, thereby determining the display style of the subtitles; the designated translation operation is a translation operation for the first language information; A subtitle display unit is used to push the subtitles to the user's video conference scene in real time according to the display style of the subtitles.

11. A subtitle processing device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images