Conference subtitle display method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a conference subtitle display method and device, electronic equipment and a readable storage medium, and the method comprises the steps: responding to a subtitle setting instruction, displaying a setting result of one or more subtitle configuration items in a client interface, and transmitting the setting result to a management server, the management server generates configuration information of the subtitles according to the setting result; obtaining text subtitle information of the current conference, and receiving configuration information sent by the management server; and performing subtitle display on the text subtitle information according to a display mode defined by the configuration information. According to the embodiment of the invention, the configuration information of the subtitles can be generated based on the setting result of one or more subtitle configuration items displayed in the client interface by the terminal, and the text subtitle information can be displayed according to the display mode defined by the configuration information, so that diversified subtitle display modes can be provided, and the user experience is improved. And better experience is brought to the user.

Description

Technical Field

[0001] The present application relates to the field of visual networking technology, and in particular to a conference subtitle display method, device, electronic device and readable storage medium. Background Art

[0002] The visual network is a real-time network that pushes many Internet applications towards high-definition video, enabling real-time transmission of high-definition video and barrier-free communication between users.

[0003] With the development of visual networking, when users participate in large-scale high-definition video conferences, relevant technologies will convert the voice information of multiple users into text information and display it in a single way in the form of subtitles on the terminals of the visual networking.

[0004] However, the subtitle display method proposed in the related art is too simple and difficult to bring a good experience to users. Summary of the invention

[0005] In view of the above problems, embodiments of the present application are proposed to provide a conference subtitle display method, device, electronic device and readable storage medium that overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, an embodiment of the present application discloses a conference subtitle display method, which is applied to a terminal, and the method includes:

[0007] In response to the subtitle setting instruction, setting results of one or more subtitle configuration items displayed in the client interface, and sending the setting results to the management server, so that the management server generates subtitle configuration information according to the setting results;

[0008] Acquire text subtitle information of the current conference, and receive the configuration information sent by the management server; the text subtitle information is: the voice information generated by the terminal in the conference, which is converted by the language conversion server to obtain text information;

[0009] The text subtitle information is displayed in subtitles according to the display mode defined by the configuration information.

[0010] In a second aspect, an embodiment of the present application discloses a conference subtitle display method, which is applied to a management server, and the method includes:

[0011] receiving a setting result sent by a terminal, wherein the setting result is obtained after the terminal operates one or more subtitle configuration items displayed in the client interface;

[0012] According to the setting result, subtitle configuration information is generated, and the configuration information is sent to the terminal, so that the terminal can display the text subtitle information corresponding to the current conference in the display mode defined by the configuration information.

[0013] In a third aspect, an embodiment of the present application discloses a conference subtitle display device, which is applied to a terminal, and the device includes:

[0014] A first processing module, configured to respond to a subtitle setting instruction, set the result of one or more subtitle configuration items displayed in the client interface, and send the setting result to a management server, so that the management server generates subtitle configuration information according to the setting result;

[0015] The first receiving module is used to obtain text subtitle information of the current conference and receive the configuration information sent by the management server; the text subtitle information is: the voice information generated by the terminal in the conference, which is converted by the language conversion server to obtain text information;

[0016] The first display module is used to display the text subtitle information in a subtitle manner according to the display mode defined by the configuration information.

[0017] In a fourth aspect, an embodiment of the present application discloses a conference subtitle display device, which is applied to a management server, and the device includes:

[0018] A second receiving module is used to receive a setting result sent by a terminal, where the setting result is obtained after the terminal operates one or more subtitle configuration items displayed in the client interface;

[0019] The second processing module is used to generate subtitle configuration information according to the setting result, and send the configuration information to the terminal, so that the terminal can display the text subtitle information corresponding to the current meeting in the display mode defined by the configuration information.

[0020] In a fifth aspect, an embodiment of the present application discloses an electronic device, characterized in that it includes:

[0021] One or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the processor to execute the method as described in any one of the first aspect or the second aspect.

[0022] In a sixth aspect, an embodiment of the present application discloses a computer-readable storage medium, characterized in that the computer program stored therein enables a processor to execute a method as described in any one of the first aspect or the second aspect.

[0023] In an embodiment of the present application, subtitle configuration information can be generated based on the setting results of one or more subtitle configuration items displayed in the client interface by the terminal, and the text subtitle information can be displayed in the subtitle mode defined by the configuration information. Therefore, a variety of subtitle display modes can be provided, and the subtitle display is more flexible, which can meet the needs of different users and bring a better experience to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of the steps of a conference subtitle display method provided by an embodiment of the present application;

[0025] Figure 2 It is a flowchart of another method for displaying conference subtitles provided by an embodiment of the present application;

[0026] Figure 3 It is a flow chart of the interactive steps of a conference subtitle display method provided by an embodiment of the present application;

[0027] Figure 4 This is an interface diagram of a conference subtitle display method provided by an embodiment of the present application;

[0028] Figure 5 This is an interface diagram of another conference subtitle display method provided by an embodiment of the present application;

[0029] Figure 6 This is a flow chart of speech-to-text conversion provided by an embodiment of the present application;

[0030] Figure 7 It is a schematic diagram of a media server connection abnormality provided by an application embodiment;

[0031] Figure 8 It is a schematic diagram of a media server network exit provided by an application embodiment;

[0032] Fig. 9 This is a flow diagram of voice-to-subtitle conversion provided by an embodiment of the present application;

[0033] Fig.10 It is a schematic diagram of a system composition provided by an embodiment of the present application;

[0034] Fig.11 It is a modular functional block diagram provided by an embodiment of the present application;

[0035] Fig.12 It is a structural block diagram of a conference subtitle display device provided in an embodiment of the application;

[0036] Fig.13 It is a structural block diagram of another conference subtitle display device provided in an embodiment of the application;

[0037] Fig.14 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0039] With the development of visual networking technology, the way of holding talks through online video has been widely accepted by the public, thus achieving instant and interactive communication. During a video conference, participants can see the images, movements, and expressions of participants in other venues, and can also hear the sounds of other venues, making participants feel as if they are in the scene. Related technologies usually convert the voice information of participants into text information and display it in the form of subtitles on the terminal of the visual network in a single way. However, this subtitle display method is too single and it is difficult to bring a good experience to users.

[0040] In an embodiment of the present application, subtitle configuration information can be generated based on the setting results of one or more subtitle configuration items displayed in the client interface by the visual network terminal, and the text subtitle information can be displayed in the subtitle mode defined by the configuration information. Therefore, a variety of subtitle display modes can be provided, and the subtitle display is more flexible, which can meet the needs of different users and bring a better experience to users.

[0041] Figure 1 This is a flowchart of a method for displaying conference subtitles provided by an embodiment of the present application, which is applied to a terminal such as Figure 1 As shown, the method may include:

[0042] Step 101: In response to a subtitle setting instruction, setting results of one or more subtitle configuration items displayed in a client interface are sent to a management server, so that the management server generates subtitle configuration information according to the setting results.

[0043] In an embodiment of the present application, the client can be installed on a visual networking terminal, and the subtitle configuration item includes some options related to subtitles, which may include: subtitle display font item, subtitle display language item, subtitle background color item, speaker nickname item, and speaker subtitle control item. In response to the subtitle setting instruction, the visual networking terminal can set one or more subtitle configuration items displayed in the client interface to obtain a setting result. Different visual networking terminals can set one or more subtitle configuration items displayed in the client interface differently, so that the needs of different users can be met.

[0044] After the visual network terminal obtains the setting result, the setting result is sent to the management server so that the management server can generate subtitle configuration information based on the setting result. Among them, the subtitle configuration information can be mode information related to subtitle display, or it can be display information related to subtitles. Since the subtitle configuration information can be generated based on the setting results of one or more subtitle configuration items displayed in the client interface by the visual network terminal, the subtitles can be displayed in the display mode defined by the configuration information. Therefore, a variety of subtitle display methods can be provided, and the subtitle display is more flexible, which can meet the needs of different users and bring a better experience to users.

[0045] Step 102: Obtain text subtitle information of the current conference and the configuration information sent by the receiving management server; the text subtitle information is: the voice information generated by the terminals in the conference, which is converted into text information by a language conversion server.

[0046] In the embodiment of the present application, the voice information generated by the visual network terminal in the conference can be converted into text information by the language conversion server, and then the language conversion server sends the text information to the media server, and after the media server receives the text information, it sends it to the visual network terminal. The embodiment of the present application can realize the function of converting voice information into text information through the interaction between the media server and the language conversion server.

[0047] Step 103: Display the text subtitle information in the display mode defined by the configuration information.

[0048] There are many display modes defined by configuration information. For example, the display mode defined by configuration information can be a line-by-line display mode. If there are participants and users A, B, and C in the meeting, the text subtitle information corresponding to the speakers can be displayed line by line in the order of speaking. For another example, the display mode defined by configuration information can be font color and font size. If the font color is set to black and the font size is set to size 4, the text subtitle information can be displayed in black and size 4.

[0049] In an embodiment of the present application, since different visual network terminals will produce different results for the settings of one or more subtitle configuration items displayed in the client interface, the configuration information of the subtitles generated based on different setting results will be different. Therefore, the text subtitle information is displayed in accordance with the display methods defined by different configuration information. A variety of subtitle display methods can be provided, and the subtitle display is more flexible, which can meet the needs of different users and bring a better experience to users.

[0050] Figure 2is a flowchart of another method for displaying conference subtitles provided by an embodiment of the present application, which is applied to a management server, such as Figure 2 As shown, the method may include:

[0051] Step 201: receiving a setting result sent by a terminal, wherein the setting result is obtained after the terminal operates one or more subtitle configuration items displayed in a client interface.

[0052] This step may refer to the above step 101 and will not be described again here.

[0053] Step 202: Generate subtitle configuration information according to the setting result, and send the configuration information to the terminal, so that the terminal can display the text subtitle information corresponding to the current conference in the display mode defined by the configuration information.

[0054] In the embodiment of the present application, after receiving the setting result sent by the terminal, the management server can generate the configuration information of the subtitle according to the setting result. Exemplarily, the management server can encrypt the setting result to obtain the configuration information of the subtitle, and encrypt the setting result to improve the security of the setting result of the subtitle configuration item.

[0055] After the terminal obtains the encrypted subtitle configuration information, it can decrypt it to obtain the decrypted subtitle configuration information, and display the text subtitle information in the display mode defined by the decrypted subtitle configuration information.

[0056] In an embodiment of the present application, subtitle configuration information can be generated based on the setting results of one or more subtitle configuration items displayed in the client interface by the terminal, and the text subtitle information can be displayed in the subtitle mode defined by the configuration information. Therefore, a variety of subtitle display modes can be provided, and the subtitle display is more flexible, which can meet the needs of different users and bring a better experience to users.

[0057] Figure 3 This is a flowchart of the interactive steps of a conference subtitle display method provided by an embodiment of the present application. Figure 3 , the step flow chart includes:

[0058] Step 301: In response to a subtitle setting instruction, the terminal sets the result of one or more subtitle configuration items displayed in the client interface, and sends the setting result to a management server, so that the management server generates subtitle configuration information according to the setting result.

[0059] This step may refer to the above step 101 and will not be described again here.

[0060] Step 302: The management server receives a setting result sent by the terminal, where the setting result is obtained after the terminal operates one or more subtitle configuration items displayed in the client interface.

[0061] This step may refer to the above step 201 and will not be described again here.

[0062] Step 303: The management server generates subtitle configuration information according to the setting result, and sends the configuration information to the terminal, so that the terminal can display the text subtitle information corresponding to the current conference in the display mode defined by the configuration information.

[0063] In an embodiment of the present application, the configuration information may include: one or more of: subtitle display position, subtitle display font, subtitle display language, subtitle background color, and speaker nickname.

[0064] In one implementation, the configuration information may include: subtitle display position. For example, the subtitle display position may include below the speaker, to the side of the speaker, and above the speaker. If the subtitle display position is below the speaker, the text subtitle information may be displayed below the speaker. If the subtitle display position is to the side of the speaker, the text subtitle information may be displayed on the left or right side of the speaker. If the subtitle display position is above the speaker, the text subtitle information may be displayed above the speaker. It should be understood that the above only lists three options, and the embodiments of the present application are not limited to this.

[0065] In another implementation, the configuration information may include: subtitle display font. For example, the subtitle display font may include: bold, Song. If the subtitle display font is bold, the text subtitle information may be displayed in bold. If the subtitle display font is Song, the text subtitle information may be displayed in Song. It should be understood that the above are only two options, and the embodiments of the present application are not limited thereto.

[0066] In another implementation, the configuration information may include: subtitle display language. For example, the subtitle display language may include: Chinese, English. If the subtitle display language is Chinese, the text subtitle information may be displayed in Chinese. If the subtitle display language is English, the text subtitle information may be displayed in English. It should be understood that the above are only two options, and the embodiments of the present application are not limited thereto.

[0067] In another implementation, the configuration information may include: subtitle background color. For example, the subtitle background color may include: black, blue, and white. The text subtitle information may be displayed in the selected subtitle background color. It should be understood that the above only lists three options, and the embodiments of the present application are not limited thereto.

[0068] In another implementation, the configuration information may include: a nickname of the speaker. For example, the nickname of the speaker may include a real name or a nickname, wherein the nickname is a non-real name and the selection of the nickname is not limited. If the speaker nickname is a real name, the real name of the speaker will be displayed in the text subtitle information so that the participants can understand the real information of the speaker. If the speaker's nickname is a nickname, for example, aaa11, then the nickname will be displayed in the text subtitle information. This method can protect the privacy of the participants. It should be understood that the above only lists two options, and the embodiments of the present application are not limited to this.

[0069] In another implementation, the configuration information may also include: speaker subtitle control. The speaker subtitle control can be understood as whether to display the subtitles of a certain speaker. For example, if there are speakers A, B, and C in the meeting, if speaker A chooses not to display the subtitles of speaker C, the subtitles of speaker C will not appear on the interface of speaker A.

[0070] It should be understood that the above-mentioned subtitle display position, subtitle display font, subtitle display language, subtitle background color, speaker nickname, and speaker subtitle control can be used in combination, and the embodiments of the present application do not make specific limitations on this.

[0071] See also Figure 4 In the seminar scheduled by user A, there are also users B, C and D. Taking the configuration information including: subtitle display language and subtitle display position as an example, if the subtitle display language is selected as Chinese, the subtitle display position is selected below the speaker, and the speaker is user A, then the text subtitle information Congratulations everyone will be displayed below user A in Chinese.

[0072] The embodiment of the present application provides multiple options for configuration information, and the configuration information is also generated based on the setting results of the terminal for one or more subtitle configuration items displayed in the client interface. Therefore, when the text subtitle information is displayed in accordance with the display method defined by the configuration information, a variety of subtitle display methods can be provided. The subtitle display is more flexible, can meet the needs of different users, and bring a better experience to users.

[0073] Step 304: The terminal obtains text subtitle information of the current conference and receives the configuration information sent by the management server; the text subtitle information is: the voice information generated by the terminal, which is converted by the language conversion server to obtain text information.

[0074] In the embodiment of the present application, the text subtitle information is configured to: if no user in the conference speaks outside the preset time range, the display of the text subtitle information is eliminated. In other words, when no user in the conference speaks outside the preset time range, the subtitle disappears.

[0075] Taking the preset time of 100 seconds as an example, if no user in the conference speaks, the display of the text subtitle information is eliminated.

[0076] In an embodiment of the present application, if the preset time range is exceeded and the users in the meeting have not spoken, the display of the text subtitle information will be eliminated. This method can save power of the device and the subtitle display is more flexible.

[0077] Step 305: The terminal displays the text subtitle information in the display mode defined by the configuration information.

[0078] In one implementation, optionally, the configuration information includes a subtitle display mode, and the subtitle display mode is a priority display mode; step 305 may include:

[0079] Sub-step 3051: when there are multiple text subtitle information generated by multiple users speaking at the same time, setting a display order for the text subtitle information corresponding to each user according to the user priority of each user;

[0080] Sub-step 3052: display the multiple text subtitle information in sequence according to the display order of the text subtitle information.

[0081] In the embodiment of the present application, user priority may also be referred to as the importance of the user. By setting a display order for the text subtitle information of the user based on the user priority, it is possible to avoid the situation where multiple users speak at the same time and the terminal program is abnormal.

[0082] For example, see Figure 5 In the seminar scheduled by user A, there are also users B, C and D. If users A, B, C and D speak at the same time, multiple text subtitle information will be generated, among which user A's text subtitle information is congratulations to everyone, user B's text subtitle information is keep up the good work, user C's text subtitle information is thank you for your help, and user D's text subtitle information is getting better and better. Since user A's priority is higher than user B's, user B's priority is higher than user C's priority, and user C's priority is higher than user D's priority. Therefore, according to the user's priority, the display order of the text subtitle information can be determined, that is, user A's text subtitle information is displayed first, then user B's text subtitle information is displayed, then user C's text subtitle information is displayed, and finally user D's text subtitle information is displayed.

[0083] In another implementation, optionally, the configuration information includes a subtitle display mode, and the subtitle display mode is a split-screen display mode; step 305 may include:

[0084] Sub-step 3053: when there are multiple participating terminals in the conference, the conference screens of the multiple terminals are displayed in split screens;

[0085] Sub-step 3054: Display the text subtitle information corresponding to the terminal currently speaking in the split-screen area corresponding to the terminal currently speaking.

[0086] For example, see Figure 4 , when there are users A, B, C and D in the conference, the conference screens of users A, B, C and D are displayed in split screens, that is, divided into four areas, where user A is in the upper left corner of the screen, user B is in the upper right corner of the screen, user C is in the lower left corner of the screen, and user D is in the lower right corner of the screen. Since the current speaker is user A, the text message "Congratulations" corresponding to user A is displayed in the split screen area corresponding to user A, that is, the split screen in the upper left corner.

[0087] In another implementation, optionally, the configuration information includes a subtitle display mode, and the subtitle display mode is a sliding display mode; step 305 may include:

[0088] Sub-step 3055: when a plurality of text subtitle information is generated in the conference, determining the number of the text subtitle information generated;

[0089] Sub-step 3056: if the generated quantity does not exceed the preset threshold, displaying the multiple text subtitle information in the same display frame in sequence according to the order of the timestamps of the text subtitle information;

[0090] Sub-step 3057: if the generated quantity exceeds the preset threshold, then in the same display frame, the text subtitle information exceeding the threshold is displayed in the order of the timestamps of the text subtitle information in the form of a scrolling control.

[0091] In the embodiment of the present application, the preset threshold is used to determine the amount of text subtitle information displayed in the same display frame. Figure 5 The scroll control is 10 in the figure, and the scroll control can be used to view at least part of the text subtitle information that is not displayed in the display box on the screen.

[0092] For example, see Figure 5 , taking the preset threshold value of 4 as an example, when multiple text subtitle information is generated in the conference, the number of text subtitle information generated is determined. If the number of generated is less than 4, multiple text subtitle information are displayed in the same display frame in the order of the timestamps of the text subtitle information. If the number of generated is more than 4, the text subtitle information exceeding the number is displayed in the same display frame in the form of a scroll control in the order of the timestamps of the text subtitle information.

[0093] Optionally, before displaying the text subtitle information in a subtitle manner according to a display mode defined by the configuration information, the method further includes:

[0094] Step 306: The terminal sends voice information to the media server, where the voice information is encrypted voice information, so that the media server can decrypt the encrypted voice information to obtain the decrypted voice information.

[0095] Step 307: The terminal receives the encrypted text information sent by the media server; the encrypted text information is: text information obtained by converting the decrypted voice information sent by the media server based on the language conversion server, and then encrypted by the media server;

[0096] Step 308: The terminal decrypts the encrypted text information to obtain the decrypted text information.

[0097] For steps 306 to 308, taking the terminal as a video network terminal as an example, see Figure 6 , the visual networking terminal sends the encrypted voice information to the media server, and the media server decrypts the encrypted voice information to obtain the decrypted voice information. The media server sends the decrypted voice information to the language conversion server, and the language conversion server converts the decrypted voice information to obtain text information, and then sends it to the media server. The media server encrypts the encrypted text information, and the media server sends the encrypted text information to the visual networking terminal, and the visual networking terminal decrypts the encrypted text information to obtain the decrypted text information. When the visual networking terminal and the media server transfer information, the encryption and decryption method is always adopted, which can improve the security of the information.

[0098] Optionally, after displaying the text subtitle information in a subtitle manner according to a display mode defined by the configuration information, the method further includes:

[0099] Step 309: the terminal performs a modification operation on one or more of the subtitle configuration items displayed in the client interface in response to the subtitle modification instruction, obtains a modified setting result, and sends the modified setting result to the management server, so that the management server generates modified configuration information of the subtitle according to the modified setting result;

[0100] Step 310: After receiving the modified configuration information sent by the management server, the terminal displays the text subtitle information in the display mode defined by the modified configuration information.

[0101] For steps 309-310, the terminal can modify one or more subtitle configuration items displayed in the client interface in response to the instruction to modify subtitles, obtain the modified setting result, and send the modified setting result to the management server. The management server generates modified configuration information of the subtitles according to the modified setting result, and sends it to the terminal. After receiving the modified configuration information sent by the management server, the terminal displays the text subtitle information in the display mode defined by the modified configuration information. The embodiment of the present application can modify one or more subtitle configuration items displayed in the client interface, so that the management server generates modified configuration information of the subtitles according to the modified setting result. Finally, the text subtitle information can be displayed in the display mode defined by the modified configuration information, which improves the flexibility of subtitle display and can meet the diverse needs of users.

[0102] Optionally, the method may further include:

[0103] Step 311: In response to a trigger operation on a preset control in the client interface, the terminal displays a subtitle record window, in which at least part of the historical text subtitle information before the current moment is displayed in the order of the timestamps of the text subtitle information.

[0104] See also Figure 5 , the preset control is 20 in the figure. If the preset control 20 in the client interface is triggered, at least part of the historical text subtitle information before the current moment can be displayed on the right side of the screen, and displayed in the order of the timestamps of the text subtitle information. For example, the historical text subtitle information includes the text subtitle information of user D. This time, thanks to everyone's joint efforts. Figure 5 There is also a whiteboard control in the text editor. By clicking on this control, the user can enter some information, which can be displayed in the subtitles based on some content. In addition, there are some basic controls, such as microphone, speaker, video, end, etc., which are not specifically limited in the embodiments of the present application.

[0105] Optionally, the method may further include:

[0106] Step 312: The management server receives the text subtitle information sent by the media server in each preset period;

[0107] Step 313: The management server generates meeting minutes based on the text subtitle information and time information of the speaker. The text subtitle information is the voice information generated by the terminal, which is converted by the language conversion server and then sent to the media server.

[0108] For step 312-step 313, the preset period can be set to any value as needed. Taking the preset period as 10 minutes as an example, the media server sends text subtitle information to the management server every 10 minutes. The management server collects statistics based on the text subtitle information and time information of the speaker and generates meeting minutes.

[0109] Optionally, the management server receives the setting result sent by the terminal through a visual network, and the method further includes:

[0110] Step 314: In response to the subtitle-enabling instruction sent by the terminal, the management server sends a message to the main media server of the visual network to obtain the idle resource number of the auxiliary media server.

[0111] Step 315: The management server receives the idle resource number of the auxiliary media server sent by the primary media server.

[0112] Step 316: The management server adds the idle auxiliary media server to the conference according to the received resource number, so that the auxiliary media server can receive the voice information sent by the terminal, and send the voice information to the language conversion server for text conversion, obtain the text subtitle information fed back after the conversion, and send the text subtitle information to the terminal.

[0113] For steps 314 to 316, the terminal sends a subtitle-enabling instruction to the management server. After receiving the instruction, the management server sends a message to the primary media server to obtain the resource number of the idle secondary media server. If the primary media server finds that there is an idle secondary media server under its service, the resource number corresponding to the idle secondary media server is sent to the management server. When the management server receives the resource number corresponding to the idle secondary media server, it invites the idle secondary media server to join the meeting according to the received resource number. After joining the meeting, the secondary media server sends a message to the management server to join the meeting and completes the meeting authentication process with the management server.

[0114] Optionally, the method may further include:

[0115] Step 317: In response to the stop subtitle instruction, the management server sends a removal instruction to the auxiliary media server, so that the auxiliary media server can perform a withdrawal operation.

[0116] Step 318: After the auxiliary media server exits the conference, the management server sends a message to the primary media server to release the resource number of the auxiliary media server, and receives a message of the release result sent by the primary media server to obtain the release result.

[0117] For step 317-step 318, if the terminal does not need the subtitle function, the subtitle function can be stopped. Figure 6 The visual network terminal sends a message to the integrated management server to stop the subtitle function. When the integrated management server receives the message to stop the subtitle function, it clears the subtitle-related flow and dynamically removes the auxiliary media server. The auxiliary media server handles the withdrawal process. After the auxiliary media server completes the withdrawal process, the integrated management server sends a message to the main media server to release the resource number of the auxiliary media server. After the main media server sends the message to release the resource number of the auxiliary media server, it returns the release result to the integrated management server.

[0118] In the embodiment of the present application, there are many situations in which the terminal sends a message to the management server to stop the subtitle function. For example, when the terminal does not need the subtitle function, the terminal sends a message to the management server to stop the subtitle function. For another example, if the media server is disconnected from the language conversion server, it can try to connect multiple times. If multiple connections fail, or if no connection is successful within a period of time, the management server sends a message to stop the subtitle function. Figure 7 , L1. The media server determines the connection status with the language conversion server; L2. If the connection between the media server and the language conversion server is disconnected, reconnect; L3. The media server can connect to the language conversion server multiple times; L4. If the media server fails to connect multiple times, a message indicating an abnormal connection with the language conversion server is sent to the integrated management server; L5. After receiving the message indicating an abnormal connection between the media server and the language conversion server, the integrated management server sends a message indicating an abnormal connection to the visual network terminal. Figure 7 The steps to turn subtitles on and off in real time can be found below. Figure 6 In another example, if the management server detects that the media server is disconnected from the network, the management server sends a message to stop the subtitle function. Figure 8 , T1, the media server logs off the network; T2, the integrated management server detects the status of the media server; T3, if the integrated management server detects that the media server logs off the network, it sends a message of the media server logging off the network to the visual network terminal, so that the media server stops the voice-to-subtitle function. Figure 8 The steps to enable subtitles and adjust the stream can be referred to as follows Figure 6 Steps S10-S11 in .

[0119] Figure 6 This is a flowchart of the interactive steps of a conference subtitle display method provided by an embodiment of the present application. Figure 6 , the process includes:

[0120] S1. After the video network conference is started, the client sends a message to set subtitle parameters to the video network terminal through transparent transmission instructions.

[0121] S2. When the visual network terminal receives the message for setting subtitle parameters, it selects and saves the subtitle parameters, and sends the result to the integrated management server. Subtitle customization settings include: subtitle position setting (below the speaker, to the side, etc.); subtitle display font selection (bold, Song, etc.); subtitle display language (Chinese, English, etc.); subtitle background color; subtitle display speaker name (real name, nickname); speaker subtitle control, whether to display the subtitles of a certain speaker. When there are multiple participants speaking at the same time, the subtitles of the chairman -> speaker 1 -> speaker 2 -> speaker 3 are displayed first. If no one speaks for a long time, the overall subtitles disappear after 10 seconds.

[0122] S3. Send a subtitle-enabling instruction to the integrated management server through the client page settings.

[0123] S4. After receiving the subtitle-enabling instruction and subtitle parameters, the integrated management server obtains the resource number corresponding to the idle streaming media server (hereinafter referred to as the auxiliary media server) from the main media server.

[0124] S5. The primary media server inquires whether there is a resource number corresponding to an idle secondary media server under its service, and if so, sends the resource number to the integrated management server.

[0125] S6. After receiving the resource number, the integrated management server invites the auxiliary media server corresponding to the resource number to join the meeting.

[0126] S7. The auxiliary media server joins the conference and sends a message of joining the conference to the integrated management server.

[0127] S8. Establish a service link with the language conversion server.

[0128] S9. The auxiliary media server and the integrated management server perform authentication of the membership process.

[0129] S10. The integrated management server sends a subtitle-enabling instruction to the video network terminal.

[0130] S11. The integrated management server handles the flow adjustment process.

[0131] S12. After turning on the subtitles, the video network terminal sends a message to the client indicating that the subtitles have been turned on successfully.

[0132] S13. During the conference, the video network terminal sends an encrypted voice stream to the auxiliary media server.

[0133] S14. After receiving the encrypted voice stream, the auxiliary media server performs audio decryption to obtain a non-encrypted voice stream, and sends the non-encrypted voice stream to the language conversion server.

[0134] S15. The language conversion server converts the non-encrypted voice stream into a non-encrypted subtitle stream, and sends the non-encrypted subtitle stream to the auxiliary media server.

[0135] S16, the auxiliary media server encrypts the non-encrypted subtitle stream to obtain the subtitle stream, and then sends it to the video network terminal.

[0136] S17. After receiving the encrypted subtitle stream, the video network terminal decrypts and decodes it, and then displays the decrypted subtitle stream.

[0137] S18. The client sends an instruction to modify subtitle parameters to the video network terminal.

[0138] S19. The auxiliary media server sends subtitle data to the integrated management server at regular intervals.

[0139] S20. The integrated management server saves the subtitle data and generates meeting minutes.

[0140] S21. The client sends an instruction to stop the subtitle function to the integrated management server.

[0141] S22. After receiving the subtitle stop instruction, the integrated management server clears the subtitle related stream.

[0142] S23. The integrated management server dynamically removes the auxiliary media server.

[0143] S24: The auxiliary media server processes the membership withdrawal process.

[0144] S25: The auxiliary media server releases the service channel.

[0145] S26. The integrated management server sends a message to the main media server to release the resource number.

[0146] S27. After the main media server releases the resource number, the release result is returned to the integrated management server.

[0147] S28. The integrated management server notifies the video network terminal to stop the real-time subtitle process.

[0148] S29. After receiving the subtitle stop instruction, the video network terminal stops subtitles and clears the subtitle display.

[0149] S30. The integrated management server returns a subtitle stop result to the client.

[0150] Fig. 9 This is a flow diagram of voice-to-subtitle conversion provided by an embodiment of the present application. Fig. 9, including: an integrated management server, a client, a visual network terminal (speaker), a visual network terminal (participant), a media server, and a language conversion server. Among them, the integrated management server, the client, the visual network terminal (speaker), and the visual network terminal (participant) all use the vehicle-to-vehicle protocol (V2V) to access the visual network. For the audio data and subtitle data in the figure, the media server sends the audio data to the language conversion server, and the language conversion server sends the subtitle data to the media server. For the speaker's voice stream and subtitle stream (voice channel) in the figure, the voice stream of the visual network terminal (speaker) is transmitted to the media server and the visual network terminal (participant) through the visual network. After receiving the voice stream, the media server interacts with the language conversion server to obtain the subtitle data, and transmits the subtitle stream to the visual network terminal (speaker) and the visual network terminal (participant) through the visual network.

[0151] Fig.10 is a schematic diagram of a system composition provided in an embodiment of the present application. Fig.10, including: integrated management server, client, visual network, media server, language conversion server, and visual network terminal. Among them, the integrated management server needs to provide the ability to access multiple sets of "media servers"; be responsible for forwarding control commands to visual network terminals; apply for media servers with idle resources and pull them into the meeting; save subtitle information and generate meeting minutes; complete the scheduling of voice streams and subtitle streams between media servers and visual network terminals. The client can specify which "media server" to use to complete the function of voice-to-subtitle conversion; can turn on and off the subtitle function, and can set the subtitle display parameters of the visual network terminal (parameters include subtitle display position, font, font size style, text color, and transparency). The visual network terminal can display and close real-time subtitles according to the instructions of the integrated management server; add audio channels to process audio and video data; can present different subtitle display positions, fonts, colors and other display effects according to the set subtitle display parameters; can parse the subtitle stream and update the display in real time. The media server is responsible for adding idle media servers to the conference and completing business authentication. It can use the V2V protocol to enter the video network. The link data on the video link side needs to be encrypted for transmission. It uses the international interconnection protocol (IP, internet protocol) to interact with the language conversion server to complete the function of voice-to-subtitle conversion. The IP side data is non-confidential communication. It can receive audio streams and decrypt and transcode them. It can complete the function of sending subtitle streams after encryption. It can complete the virtual terminal settings for network access and the language conversion server parameter settings through the configuration file. It uses the main message number to interact with the integrated management server for control commands and uses the resource number to provide voice-to-subtitle business services. There needs to be a clear prompt when resources are insufficient. It needs to support the concurrent capability of at least 20 meetings to start voice-to-subtitle conversion at the same time. The number of concurrent meetings is configurable and can be adjusted according to the authorized number of language conversion servers.

[0152] Fig.11 This is a modular functional block diagram provided by the embodiment of the present application, see Fig.11 , including: conference venue and core computer room, the data generated by both are transmitted through the visual network. Among them, the conference venue includes a comprehensive management client and terminal, and the terminal can include a microphone, a high-definition camera, and a TV. The core computer room includes a voice system, a media service system, a cryptographic system, a system management system, and a recording and broadcasting service system. Among them, the voice system is responsible for converting the voice information in the meeting into subtitle information, the media service system is used to receive the voice information sent by the terminal, and interact with the voice system to obtain subtitle information, the cryptographic system is used to encrypt or decrypt the voice information and subtitle information in the meeting, and the video conferencing system is responsible for processing the video images in the meeting.

[0153] See also Fig.12 The embodiment of the present application provides a conference subtitle display device 40, and the conference subtitle display device 40 includes:

[0154] The first processing module 401 is used to respond to the subtitle setting instruction, set the result of one or more subtitle configuration items displayed in the client interface, and send the setting result to the management server, so that the management server generates subtitle configuration information according to the setting result;

[0155] The first receiving module 402 is used to obtain text subtitle information of the current conference and receive the configuration information sent by the management server; the text subtitle information is: the voice information generated by the terminal in the conference, which is converted by the language conversion server to obtain text information;

[0156] The first display module 403 is used to display the text subtitle information in a subtitle manner according to the display mode defined by the configuration information.

[0157] Optionally, the configuration information includes a subtitle display mode, and the subtitle display mode is a priority display mode;

[0158] The first display module 403 includes:

[0159] A setting submodule, used to set a display order for the text subtitle information corresponding to each user according to the user priority of each user when there are multiple text subtitle information generated by multiple users speaking at the same time;

[0160] The first display submodule is used to display the plurality of text subtitle information in sequence according to the display order of the text subtitle information.

[0161] Optionally, the configuration information includes a subtitle display mode, and the subtitle display mode is a split-screen display mode; and the device further includes:

[0162] A split-screen display module is used to display the conference screens of multiple video networking terminals in split screens when there are multiple video networking terminals participating in the conference;

[0163] The first display module 403 includes:

[0164] The second display submodule is used to display the text subtitle information corresponding to the visual networking terminal that is currently speaking in the split screen area corresponding to the visual networking terminal that is currently speaking.

[0165] Optionally, the configuration information includes a subtitle display mode, and the subtitle display mode is a sliding display mode;

[0166] The first display module 403 includes:

[0167] A determination submodule, used for determining the number of generated text subtitle information when multiple text subtitle information is generated in a conference;

[0168] A third display submodule, configured to display the plurality of text subtitle information in sequence in the same display frame according to the order of the timestamps of the text subtitle information if the generated quantity does not exceed a preset threshold;

[0169] The fourth display submodule is used to display the text subtitle information exceeding the preset threshold in the same display frame in the form of a scrolling control in the order of the timestamps of the text subtitle information if the generated quantity exceeds the preset threshold.

[0170] Optionally, the device further comprises:

[0171] The second display module is used to display a subtitle record window in response to a trigger operation on a preset control in the client interface, in which at least part of the historical text subtitle information before the current moment is displayed in the order of the timestamps of the text subtitle information.

[0172] In an embodiment of the present application, subtitle configuration information can be generated based on the setting results of one or more subtitle configuration items displayed in the client interface by the terminal, and the text subtitle information can be displayed in the subtitle mode defined by the configuration information. Therefore, a variety of subtitle display modes can be provided, and the subtitle display is more flexible, which can meet the needs of different users and bring a better experience to users.

[0173] See also Fig.13 , the embodiment of the present application provides a conference subtitle display device 50, the conference subtitle display device 50 comprising:

[0174] The second receiving module 501 is used to receive the setting result sent by the terminal, where the setting result is obtained after the terminal operates one or more subtitle configuration items displayed in the client interface;

[0175] The second processing module 502 is used to generate subtitle configuration information according to the setting result, and send the configuration information to the terminal, so that the terminal can display the text subtitle information corresponding to the current conference in the display mode defined by the configuration information.

[0176] Optionally, the management server receives the setting result sent by the terminal through a visual network, and the device further includes:

[0177] A first sending module, configured to send a message for obtaining an idle resource number of the auxiliary media server to the main media server of the visual network in response to a subtitle opening instruction sent by the terminal;

[0178] A third receiving module, configured to receive the idle resource number of the auxiliary media server sent by the primary media server;

[0179] The third processing module is used to add the idle auxiliary media server to the conference according to the received resource number, so that the auxiliary media server can receive the voice information sent by the terminal, send the voice information to the language conversion server for text conversion, obtain text subtitle information fed back after conversion, and send the text subtitle information to the terminal;

[0180] In an embodiment of the present application, subtitle configuration information can be generated based on the setting results of one or more subtitle configuration items displayed in the client interface by the terminal, and the text subtitle information can be displayed in the subtitle mode defined by the configuration information. Therefore, a variety of subtitle display modes can be provided, and the subtitle display is more flexible, which can meet the needs of different users and bring a better experience to users.

[0181] The embodiment of the present invention discloses an electronic device, such as Fig.14 As shown, it includes: a processor 1401 and a memory 1402, the memory 1402 stores programs or instructions that can be run on the processor 1401, and when the program or instruction is executed by the processor 1401, it implements the dynamic access control method based on the zero trust model as described in any one of the above items.

[0182] An embodiment of the present invention discloses a computer-readable storage medium, wherein a computer program stored in the storage medium enables a processor to execute any of the above methods.

[0183] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0184] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0189] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0190] The above is a detailed introduction to a conference subtitle display method, device, equipment and storage medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A conference subtitle display method, applied to a terminal, characterized in that: include: In response to the subtitle setting instruction, setting results of one or more subtitle configuration items displayed in the client interface, and sending the setting results to the management server, so that the management server generates subtitle configuration information according to the setting results; Acquire text subtitle information of the current conference, and receive the configuration information sent by the management server; the text subtitle information is: the voice information generated by the terminal in the conference, which is converted by the language conversion server to obtain text information; The text subtitle information is displayed in subtitles according to the display mode defined by the configuration information.

2. The method according to claim 1, characterized in that: The configuration information includes a subtitle display mode, and the subtitle display mode is a priority display mode; The step of displaying the text subtitle information in a subtitle manner according to the display mode defined by the configuration information includes: When there are multiple text subtitle information generated by multiple users speaking at the same time, a display order is set for the text subtitle information corresponding to each user according to the user priority of each user; According to the display order of the text subtitle information, multiple text subtitle information are displayed in sequence.

3. The method according to claim 1, characterized in that The configuration information includes a subtitle display mode, and the subtitle display mode is a split-screen display mode; the method further includes: When there are multiple terminals participating in the conference, the conference screens of the multiple terminals are displayed on split screens; The step of displaying the text subtitle information in a subtitle manner according to the display mode defined by the configuration information includes: The text subtitle information corresponding to the terminal currently speaking is displayed in the split-screen area corresponding to the terminal currently speaking.

4. The method according to claim 1, characterized in that The configuration information includes a subtitle display mode, and the subtitle display mode is a sliding display mode; The step of displaying the text subtitle information in a subtitle manner according to the display mode defined by the configuration information includes: When a plurality of text subtitle information is generated in a conference, determining the number of the text subtitle information generated; If the generated quantity does not exceed the preset threshold, displaying the multiple text subtitle information in the same display frame in sequence according to the order of the timestamps of the text subtitle information; If the generated quantity exceeds the preset threshold, the text subtitle information exceeding the threshold is displayed in the same display frame in the form of a scrolling control in the order of the timestamps of the text subtitle information.

5. The method according to claim 1, characterized in that The method further comprises: In response to a triggering operation on a preset control in the client interface, a subtitle record window is displayed, in which at least part of the historical text subtitle information before the current moment is displayed in the order of the timestamps of the text subtitle information.

6. A conference subtitle display method, applied to a management server, characterized in that: include: receiving a setting result sent by a terminal, wherein the setting result is obtained after the terminal operates one or more subtitle configuration items displayed in the client interface; According to the setting result, subtitle configuration information is generated, and the configuration information is sent to the terminal, so that the terminal can display the text subtitle information corresponding to the current conference in the display mode defined by the configuration information.

7. The subtitle display method according to claim 6, characterized in that: The management server receives the setting result sent by the terminal through the visual network, and the method further includes: In response to the subtitle-enabling instruction sent by the terminal, a message for obtaining an idle secondary media server resource number is sent to the primary media server of the visual network; Receiving the idle resource number of the auxiliary media server sent by the primary media server; According to the received resource number, the idle auxiliary media server is added to the conference so that the auxiliary media server can receive the voice information sent by the terminal, and send the voice information to the language conversion server for text conversion, obtain the text subtitle information fed back after the conversion, and send the text subtitle information to the terminal.

8. A conference subtitle display device, applied to a terminal, characterized in that: The device comprises: A first processing module, configured to respond to a subtitle setting instruction, set the result of one or more subtitle configuration items displayed in the client interface, and send the setting result to a management server, so that the management server generates subtitle configuration information according to the setting result; The first receiving module is used to obtain text subtitle information of the current conference and receive the configuration information sent by the management server; the text subtitle information is: the voice information generated by the terminal in the conference, which is converted by the language conversion server to obtain text information; The first display module is used to display the text subtitle information in a subtitle manner according to the display mode defined by the configuration information.

9. A conference subtitle display device, applied to a management server, characterized in that: The device comprises: A second receiving module is used to receive a setting result sent by a terminal, where the setting result is obtained after the terminal operates one or more subtitle configuration items displayed in the client interface; The second processing module is used to generate subtitle configuration information according to the setting result, and send the configuration information to the terminal, so that the terminal can display the text subtitle information corresponding to the current meeting in the display mode defined by the configuration information.

10. An electronic device, characterized in that: include: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the processors to perform the method of any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the method according to any one of claims 1 to 7.

Images