Multimedia data transmission method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a multimedia data transmission method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: responding to a received instruction for starting multimedia data translation, and obtaining target multimedia data in a first terminal; and under the condition that the first terminal has the translation capability, translating the target multimedia data to obtain translated content, and sending the translated content to the second terminal for displaying on the second terminal. Therefore, in a multi-user communication scene, if languages among multiple users are not communicated, an AI translation circulation function can be started, so that corresponding translation contents can be displayed on client equipment needing translation while communication is performed through host equipment. Therefore, the user needing to translate can know the specific content of communication in time, so that the smooth progress of the communication process can be ensured, and the efficiency of the communication can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a method, device, electronic device, storage medium and computer program product for transmitting multimedia data. Background Art

[0002] With the continuous advancement of globalization, the demand for cross-language communication is growing. In multinational companies, colleagues with different language and cultural backgrounds often communicate with each other. At this time, multiple users with different language backgrounds will have the need for content translation.

[0003] For example, multiple users with the same language background may share a document through the screen sharing function. The document is in a language that all these users understand, and these users will communicate with each other in the language they understand. At this time, if a user with another language background also wants to join the communication, since the user does not understand the language of the current users, he will not be able to understand the specific content of the communication in time, which will hinder the communication process and seriously affect the efficiency of communication. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device, storage medium and computer program product for transmitting multimedia data, so as to at least solve the problem in the above-mentioned related technologies that if multiple users do not speak the same language, the communication process will be hindered, which will seriously affect the efficiency of communication.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for transmitting multimedia data is provided, which is applied to a first terminal, and the first terminal communicates multimedia data with a second terminal, including: in response to receiving an instruction to start multimedia data translation, obtaining target multimedia data in the first terminal; when the first terminal has translation capability, translating the target multimedia data to obtain translated content, and sending the translated content to the second terminal for display on the second terminal.

[0006] Optionally, the method further includes: when the first terminal does not have translation capability and the second terminal has translation capability, sending the target multimedia data and a translation instruction to the second terminal, wherein the translation instruction is used to instruct the second terminal to translate the target multimedia data.

[0007] Optionally, it also includes: when neither the first terminal nor the second terminal has translation capability, sending the target multimedia data and translation instructions to a third-party device with translation capability, wherein the translation instructions are used to instruct the third-party device to translate the target multimedia data, and the obtained translation content is sent to the second terminal.

[0008] Optionally, the method further includes: querying whether the first terminal and / or the second terminal has translation capability; and determining a main device for translating the target multimedia data according to the query result.

[0009] Optionally, the querying whether the second terminal has the translation capability includes: sending a translation capability query request to the second terminal; and receiving information about the translation capability of the second terminal from the second terminal.

[0010] Optionally, sending the translated content to the second terminal includes: projecting the translated content to the second terminal.

[0011] Optionally, the target multimedia data includes screen content of the first terminal and / or subtitle content obtained by performing voice recognition on audio content of the first terminal.

[0012] Optionally, it also includes: displaying original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; wherein the translation content corresponding to the original content is used for display on the second terminal.

[0013] Optionally, it also includes: displaying original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; sending the original content to the second terminal, wherein the original content and translated content corresponding to the original content are used for display on the second terminal.

[0014] Optionally, it also includes: displaying original content and translated content corresponding to the original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; sending the original content to the second terminal, wherein the original content and the translated content corresponding to the original content are used for display on the second terminal.

[0015] Optionally, in the case that the original content includes the subtitle content, the subtitle content is not displayed on the first terminal, and the subtitle content and the translation content corresponding to the subtitle content are used to be displayed on the second terminal. According to a second aspect of an embodiment of the present disclosure, a method for transmitting multimedia data is provided, which is applied to a second terminal, and the second terminal communicates multimedia data with a first terminal, comprising: receiving translation content for target multimedia data in the first terminal from the first terminal when the first terminal has translation capability, wherein the translation content is obtained by the first terminal translating the target multimedia data in response to receiving an instruction to start multimedia data translation; and displaying the translation content.

[0016] Optionally, the method further includes: when the first terminal does not have translation capability and the second terminal has translation capability, receiving the target multimedia data and a translation instruction from the first terminal; and translating the target multimedia data in response to the translation instruction to obtain translation content.

[0017] Optionally, the method further includes: when neither the first terminal nor the second terminal has translation capability, receiving translation content for the target multimedia data, wherein the translation content is obtained by translating the target multimedia data by a third-party device with translation capability.

[0018] Optionally, the method further includes: receiving a translation capability query request from the first terminal; and sending information about the translation capability of the second terminal to the first terminal in response to the translation capability query request.

[0019] Optionally, receiving, from the first terminal, the translated content for the target multimedia data in the first terminal includes: receiving the translated content projected from the first terminal.

[0020] Optionally, the target multimedia data includes screen content of the first terminal and / or subtitle content obtained by performing voice recognition on audio content of the first terminal.

[0021] Optionally, displaying the translated content includes: displaying the translated content corresponding to the original content on the second terminal, wherein the original content includes the screen content and / or the subtitle content; wherein the original content is also displayed on the first terminal.

[0022] Optionally, it also includes: receiving original content from the first terminal, wherein the original content includes the screen content and / or the subtitle content, and the original content is also displayed on the first terminal; wherein displaying the translated content includes: displaying the original content and the translated content corresponding to the original content on the second terminal.

[0023] Optionally, the original content and the translated content corresponding to the original content are used for display on the first terminal, wherein the original content includes the screen content and / or the subtitle content; and further includes: receiving the original content from the first terminal; and displaying the translated content includes: displaying the original content and the translated content corresponding to the original content on the second terminal.

[0024] Optionally, when the original content includes the subtitle content, displaying the original content and the translation content corresponding to the original content on the second terminal includes: displaying the subtitle content and the translation content corresponding to the subtitle content on the second terminal; wherein the subtitle content is not displayed on the first terminal.

[0025] According to a third aspect of an embodiment of the present disclosure, a multimedia data transmission device is provided, which is applied to a first terminal, and the first terminal communicates multimedia data with a second terminal, including: a multimedia data acquisition module, configured to acquire target multimedia data in the first terminal in response to receiving an instruction to start multimedia data translation; a translation module, configured to translate the target multimedia data when the first terminal has translation capability, obtain translation content, and send the translation content to the second terminal for display on the second terminal.

[0026] Optionally, the transmission device also includes: a first multimedia data sending module, configured to send the target multimedia data and a translation instruction to the second terminal when the first terminal does not have translation capability and the second terminal has translation capability, wherein the translation instruction is used to instruct the second terminal to translate the target multimedia data.

[0027] Optionally, the transmission device also includes: a second multimedia data sending module, configured to send the target multimedia data and translation instructions to a third-party device with translation capabilities when neither the first terminal nor the second terminal has translation capabilities, wherein the translation instructions are used to instruct the third-party device to translate the target multimedia data, and the obtained translation content is sent to the second terminal.

[0028] Optionally, the transmission device also includes: a query module configured to query whether the first terminal and / or the second terminal has translation capabilities; a main device determination module configured to determine the main device used to translate the target multimedia data based on the query result.

[0029] Optionally, the query module is configured to: send a translation capability query request to the second terminal; and receive information about the translation capability of the second terminal from the second terminal.

[0030] Optionally, the translation module is configured to: project the translated content to the second terminal.

[0031] Optionally, the target multimedia data includes screen content of the first terminal and / or subtitle content obtained by performing voice recognition on audio content of the first terminal.

[0032] Optionally, the transmission device also includes: a first original content display module, configured to display original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; wherein the translation content corresponding to the original content is used to be displayed on the second terminal.

[0033] Optionally, the transmission device also includes: a second original content display module, configured to display original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; a first original content sending module, configured to send the original content to the second terminal, wherein the original content and the translated content corresponding to the original content are used for display on the second terminal.

[0034] Optionally, the transmission device also includes: an original content and translated content display module, configured to display original content and translated content corresponding to the original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; a second original content sending module, configured to send the original content to the second terminal, wherein the original content and translated content corresponding to the original content are used for display on the second terminal.

[0035] Optionally, in the case that the original content includes the subtitle content, the subtitle content is not displayed on the first terminal, and the subtitle content and the translation content corresponding to the subtitle content are used to be displayed on the second terminal. According to a fourth aspect of an embodiment of the present disclosure, a multimedia data transmission device is provided, which is applied to a second terminal, and the second terminal communicates multimedia data with a first terminal, including: a first translation content receiving module, configured to receive translation content for target multimedia data in the first terminal from the first terminal when the first terminal has translation capability, wherein the translation content is obtained by the first terminal translating the target multimedia data in response to receiving an instruction to start multimedia data translation; and a translation content display module, configured to display the translation content.

[0036] Optionally, the transmission device also includes: a multimedia data receiving module, configured to receive the target multimedia data and a translation instruction from the first terminal when the first terminal does not have translation capability and the second terminal has translation capability; a translation module, configured to translate the target multimedia data in response to the translation instruction to obtain translation content.

[0037] Optionally, the transmission device also includes: a second translation content receiving module, configured to receive translation content for the target multimedia data when neither the first terminal nor the second terminal has translation capability, wherein the translation content is obtained by translating the target multimedia data by a third-party device with translation capability.

[0038] Optionally, the transmission device also includes: a query request receiving module configured to receive a translation capability query request from the first terminal; and a translation capability information sending module configured to send information about the translation capability of the second terminal to the first terminal in response to the translation capability query request.

[0039] Optionally, the first translation content receiving module is configured to: receive the translation content projected from the first terminal.

[0040] Optionally, the target multimedia data includes screen content of the first terminal and / or subtitle content obtained by performing voice recognition on audio content of the first terminal.

[0041] Optionally, the translation content display module is configured to: display translation content corresponding to original content on the second terminal, wherein the original content includes the screen content and / or the subtitle content; wherein the original content is also displayed on the first terminal.

[0042] Optionally, the transmission device also includes: a first original content receiving module, configured to receive original content from the first terminal, wherein the original content includes the screen content and / or the subtitle content, and the original content is also displayed on the first terminal; the translation content display module is configured to: display the original content and the translation content corresponding to the original content on the second terminal.

[0043] Optionally, the original content and the translated content corresponding to the original content are used to be displayed on the first terminal, wherein the original content includes the screen content and / or the subtitle content; the transmission device also includes: a second original content receiving module, configured to receive the original content from the first terminal; the translated content display module is configured to: display the original content and the translated content corresponding to the original content on the second terminal.

[0044] Optionally, when the original content includes the subtitle content, the translation content display module is configured to: display the subtitle content and translation content corresponding to the subtitle content on the second terminal; wherein the subtitle content is not displayed on the first terminal.

[0045] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method for transmitting multimedia data according to the present disclosure.

[0046] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for transmitting multimedia data according to the present disclosure.

[0047] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for transmitting multimedia data according to the present disclosure is implemented.

[0048] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: In the present disclosure, in a multi-user communication scenario, if the multiple users do not speak the same language, the AI ​​translation function can be enabled, so that while communicating through the host device, the corresponding translation content can be displayed on the client device that needs translation. In this way, users who need translation can understand the specific content of the communication in a timely manner, and then ensure that the communication process is smooth, thereby improving the efficiency of communication.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0051] Figure 1 is a schematic diagram showing a scenario in which multiple users with different language and cultural backgrounds conduct an online meeting in the related art; Figure 2 is a schematic diagram showing a translation flow architecture according to an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram showing a scenario of using a translation flow function in an online conference according to an exemplary embodiment of the present disclosure; Figure 4 is a flowchart illustrating a method for transmitting multimedia data according to an exemplary embodiment of the present disclosure; Figure 5 is a schematic diagram showing a pop-up flow floating window when entering a specific interface of a third-party app according to an exemplary embodiment of the present disclosure; Figure 6 is a schematic diagram showing a plurality of nearby devices searched by a host device for AI translation flow according to an exemplary embodiment of the present disclosure; Figure 7 is a schematic diagram showing a flow chart of audio translation flow on a host side according to an exemplary embodiment of the present disclosure; Figure 8 is a schematic diagram showing a flow chart of screen content translation flow on a host side according to an exemplary embodiment of the present disclosure; Fig. 9 is a schematic diagram showing a flow chart of audio translation flow on a client side according to an exemplary embodiment of the present disclosure; Fig.10 is a schematic diagram showing a flow chart of screen translation flow on a client side according to an exemplary embodiment of the present disclosure; Fig.11 is a schematic diagram showing the effect of transferring the AI ​​translation flow to a tablet according to an exemplary embodiment of the present disclosure; Fig.12 is a schematic diagram showing the effect of streaming AI translation to a TV according to an exemplary embodiment of the present disclosure; Fig.13 is a schematic diagram showing that only the Host end has AI translation capability according to an exemplary embodiment of the present disclosure; Fig.14 is a schematic diagram showing the effect of projecting the translated content to the Client device using the projection protocol according to an exemplary embodiment of the present disclosure; Fig.15 is a schematic diagram showing displaying translated screen content on a second terminal according to an exemplary embodiment of the present disclosure; Fig.16 is a schematic diagram showing displaying translated Korean screen content, original Chinese subtitle content, and translated Korean subtitle content on a second terminal according to an exemplary embodiment of the present disclosure; Fig.17 is a schematic diagram showing displaying original subtitle content and translated subtitle content on a second terminal according to an exemplary embodiment of the present disclosure; Fig.18 is a flow chart illustrating another method for transmitting multimedia data according to an exemplary embodiment of the present disclosure; Fig.19 is a block diagram showing a transmission device of multimedia data according to an exemplary embodiment of the present disclosure; Fig. 20 is a block diagram showing another multimedia data transmission device according to an exemplary embodiment of the present disclosure; Fig.21 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0054] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0055] With the continuous advancement of globalization, the demand for cross-language communication is growing. As an important tool to connect different cultures and promote international cooperation, the development level of translation technology directly affects the efficiency and quality of global information circulation. Although current translation tools can provide basic translation services, in multi-person conference scenarios, when there are participants with different language and cultural backgrounds, a single translation tool may not be able to meet the needs of multiple participants with different language and cultural backgrounds for linkage and synchronization.

[0056] Figure 1 is a schematic diagram showing a scenario in which multiple users with different language and cultural backgrounds conduct an online meeting in the related art. Figure 1 , suppose that in an online video conference, Chinese colleague A and remote Chinese client C share a Chinese document through the conference screen sharing function, and communicate about the document in Chinese. During the video conference, a problem is encountered and needs to be confirmed by Korean colleague B. However, Korean colleague B's Chinese is not very good, and he cannot understand the content of the shared Chinese document and the content that Chinese colleague A and Chinese client C are communicating about. This will hinder the communication process and seriously affect the efficiency of communication.

[0057] In order to solve the above problems existing in the related art, the multimedia data transmission method, device, electronic device, storage medium and computer program product provided by the present disclosure can enable the AI ​​translation flow function in a multi-user communication scenario if the multiple users do not speak the same language, so that while communicating through the host device, the corresponding translation content can be displayed on the client device that needs translation. In this way, users who need translation can understand the specific content of the communication in a timely manner, and then ensure that the communication process is smoothly advanced, thereby improving the efficiency of communication.

[0058] Figure 2 is a schematic diagram showing a translation flow architecture according to an exemplary embodiment of the present disclosure.

[0059] Reference Figure 2, showing the host side and the client side. The host side can be but not limited to: a smartphone (phone), a tablet (tablet), etc.; the client side can be but not limited to: a smartphone, a tablet, a personal computer (Personal Computer, PC), a television (Television, TV), etc. In addition, both the host side and the client side can include (VCC App), a screen content translation module (Screen Translation / OCR), an audio transmission module (AudioTransfer) and a voice recognition module (VOIP ASR), etc. In addition, in addition to the above modules, the host side can also include a third-party application (3rd App), and the third-party application can be a video call application.

[0060] Specifically, VCC App is mainly used to provide streaming functions for third-party video call apps.

[0061] Audio Transfer is mainly used to implement AI audio translation control logic, which can specifically include: the host side requests to query the AI ​​audio translation capabilities of the client side; the host side transmits the results of AI audio translation to the client side; the host side controls the language switching capabilities of the AI ​​audio translation of the client side, etc.

[0062] OCR / Screen Translation is mainly used to control OCR recognition and translation logic. That is, OCR / Screen Translation can first use OCR technology to perform character recognition on the screen content, and then translate the recognized character content. Then, Open CV can also be used to erase the original screen content and fill the erased position with the translated content.

[0063] For 3rd app: When entering the video call interface or online meeting interface of the 3rd app, a video streaming floating window option may appear, and users can use VCC App to stream by clicking on the option.

[0064] Audio hal is mainly used to implement audio ASR and translation functions, and can provide relevant interfaces for VCCApp to call. The Data bus is mainly used for data transmission between the host and the client. For example, the AI ​​translation result of the host can be sent to the client through the Data bus.

[0065] The Control bus is mainly used for the exchange of basic information and the transmission of control commands. For example, the host can send a request to the client through the Control bus to query whether the client has the AI ​​audio translation capability, and then the client can return specific capability information to the host through the Control bus; or the host can also send the language setting command of the AI ​​audio translation to the client through the Control bus, etc.

[0066] It should be noted that, in the present disclosure, only the host side may include the Screen Translation / OCR module and the VOIP ASR module; or, only the client side may include the Screen Translation / OCR module and the VOIP ASR module; or, both the host side and the client side may include the Screen Translation / OCR module and the VOIP ASR module. The present disclosure does not make any specific restrictions on this, and the aforementioned implementation is merely an exemplary description.

[0067] Figure 3 2 is a schematic diagram showing a scenario of using a translation flow function in an online conference according to an exemplary embodiment of the present disclosure. Figure 3 , Chinese colleague A can synchronize the document content on his terminal (Host) to the device (Client) of Korean colleague B. At this time, Chinese colleague A's terminal can still display Chinese documents, and Chinese colleague A can still communicate with Chinese customer C in Chinese; while Korean colleague B's device can display the Korean document content obtained by translating the Chinese document, and Korean colleague B's device can also display the original conversation subtitles and translated subtitles between Chinese colleague A and Chinese customer C. In this way, even if Korean colleague B does not understand Chinese, he can still understand the specific content of the communication between the participating parties in a timely manner, which can ensure the smooth progress of the communication process and improve the efficiency of communication.

[0068] In the present disclosure, by combining the flow function of video conferencing applications between multiple devices with the AI ​​translation function, it is possible to help multi-person meetings with different language backgrounds achieve a smooth communication experience.

[0069] Figure 4 The present invention is a flowchart showing a method for transmitting multimedia data according to an exemplary embodiment of the present disclosure. The method is applied to a first terminal, and the first terminal and a second terminal can perform multimedia data communication.

[0070] Reference Figure 4In step 401, in response to receiving an instruction to start multimedia data translation, target multimedia data in the first terminal can be obtained.

[0071] Specifically, it can be pre-configured that when a user enters a specific interface of a third-party app, the VCC App on the host device is controlled to pop up a flow floating window on the specific interface, and the specific interface can be but is not limited to: a video call interface, an online meeting interface, an audio conference interface, etc. Figure 5 It is a schematic diagram showing a pop-up flow floating window when entering a specific interface of a third-party app according to an exemplary embodiment of the present disclosure.

[0072] Next, users can click Figure 5 Specifically, when the user clicks Figure 5 When the transfer floating window pops up in the translation window, the host device can search for surrounding devices through BLE or wifi P2P. Then, the user can select a device from the multiple devices searched, and then start the AI ​​translation transfer function. In addition, the user can also change the target language of the translation through settings. Figure 6 is a schematic diagram showing a plurality of nearby devices searched by a host device for AI translation flow according to an exemplary embodiment of the present disclosure.

[0073] According to an exemplary embodiment of the present disclosure, the target multimedia data may include the screen content of the first terminal and / or the subtitle content obtained by performing voice recognition on the audio content of the first terminal. The audio content may be the speech content of the participants during the communication process, or the audio file referenced by the shared video material; the screen content may be the shared document content, video file, PPT content, etc.

[0074] In step 402, if the first terminal has translation capability, the target multimedia data can be translated to obtain translation content, and the translation content can be sent to the second terminal for display on the second terminal. That is, the above VCC App can perform translation transfer according to the AI ​​capabilities of the host and the client, including translation transfer of audio content and / or translation transfer of screen content.

[0075] Specifically, when users are holding online meetings (for example, video conferences or voice conferences), the host device is a participant in the meeting, and the client device does not directly participate in the meeting. After the AI ​​translation flow function is enabled, the screen content shared on the host device and the speeches of the participants can be translated into the specified language and displayed on the client device. In addition, depending on whether the host device and the client device support ASR and OCR capabilities, the translation operation may be completed on the host device or on the client device.

[0076] According to an exemplary embodiment of the present disclosure, the target multimedia data may be audio content. Speech recognition may be performed on the audio content. Exemplarily, ASR technology may be used to perform speech recognition on the audio content to obtain subtitle content corresponding to the audio content, i.e., text content. Then, the text content may be translated into text content in the target language.

[0077] In this way, by translating the audio content generated during the meeting into text content in a specified language, it is convenient for users on the client side who do not speak the language to timely understand the specific content of the communication between the participating users, thereby ensuring the smooth progress of the meeting and improving the efficiency of communication.

[0078] According to an exemplary embodiment of the present disclosure, it is also possible to query whether the first terminal and / or the second terminal has translation capability. Then, the main device for translating the target multimedia data can be determined according to the query result.

[0079] According to an exemplary embodiment of the present disclosure, the first terminal may send a translation capability query request to the second terminal. Then, the first terminal may receive information about the translation capability of the second terminal from the second terminal.

[0080] According to an exemplary embodiment of the present disclosure, when the first terminal has a translation capability, the first terminal may translate the target multimedia data to obtain translation content.

[0081] For the audio content, you can determine whether to use the Host device or the Client device for audio content translation and transfer based on whether the Host device and the Client device support ASR capabilities. Specifically, if both the Host device and the Client device have ASR capabilities, the Host device can be used for ASR and translation first; if only the Host device has ASR capabilities and the Client device does not have ASR capabilities, the Host device can be used for ASR and translation.

[0082] Figure 7It is a schematic diagram showing the flow of audio translation flow on the host side according to an exemplary embodiment of the present disclosure.

[0083] Reference Figure 7 In step 701, after the host starts the audio translation function, the host can obtain the audio content output by the speaker and the audio content collected by the microphone. Exemplarily, the audio content output by the speaker can include the speech content of the other party's participating user and the audio file content referenced by the shared video material; the audio content collected by the microphone can be the speech content of the user on the host side and / or the speech content of the user on the client side.

[0084] In step 702, the host can perform speech recognition ASR on the audio content output by the speaker and the audio content collected by the microphone to obtain text content. It should be noted that the audio content collected by the microphone of the host can be the speech of the host user or the speech of the client user, and they may speak different languages. In addition, the ASR system has multi-language recognition capabilities, and can automatically recognize audio in multiple languages ​​to obtain corresponding text content.

[0085] In step 703, the host side can translate the text content obtained by voice recognition into text content in a specified language. Specifically, when the speech content of the user on the host side is collected through the microphone, the speech content of the user on the host side can be translated into the content of the language familiar to the user on the client side; or, when the speech content of the user on the client side is collected through the microphone, the speech content of the user on the client side can be translated into the content of the language familiar to the user on the host side. In this way, it can be ensured that each participant can know the speech content of other parties in a timely manner, thereby ensuring the smooth progress of the meeting process.

[0086] In step 704, the host may send the original text content obtained through speech recognition and the text content in the specified language obtained through translation to the client device.

[0087] In step 705, the client device may display the original text content obtained through speech recognition and the text content in the specified language obtained through translation. Specifically, the original speech content of the user at the host end and the corresponding translation content, the original speech content of the user at the client end and the corresponding translation content, the original speech content of the other party's participating user and the corresponding translation content, and the original text content corresponding to the audio file referenced by the shared video material and the corresponding translation content may be displayed in different areas on the screen of the client device, so as to avoid interference between speech recognition contents corresponding to different audio output parties.

[0088] For the screen content, before translating the screen content, the host can send a command to the client to inquire whether the client has the "OCR+translation" capability. Then, based on the actual situation of whether the host and client devices support "OCR+translation", it can be determined whether to use the host device or the client device for screen translation. Specifically, if both the host and client devices support OCR and translation capabilities; or if only the host has OCR and translation capabilities, but the client does not, the "OCR+translation" function of the host device can be called to perform character recognition and translation on the screen content currently displayed on the Host device, and the translated content can be transmitted to the Client device for display through the VCC App.

[0089] Figure 8 It is a schematic diagram showing the flow of screen content translation flow on the host side according to an exemplary embodiment of the present disclosure.

[0090] Reference Figure 8 In step 801, the host can obtain the video stream of the host screen through the virtual screen, and can capture the bitmap of the current frame of the video stream every 2 seconds. Then, the two bitmaps captured before and after can be compared to see if they have changed. If the bitmap has not changed, return to step 801; if the bitmap has changed, execute step 802.

[0091] In step 802, if the bitmap changes, it means that a video frame switch has occurred, that is, the host screen currently switches to display a new video frame. At this time, the host can trigger an OCR instruction to extract the character content to be translated from the current video frame.

[0092] In step 803, the host may translate the character content extracted from the video frame into character content in a specified language.

[0093] In step 804, the host can use open Cv to erase the original character content on the video frame, and can use the character content of the specified language obtained by translation to fill the erased position of the video frame. Then, the video frame after the character content is replaced can be saved.

[0094] In step 805, the host may send the translated video frame to the client. It should be noted that in the present disclosure, the host device may send only the translated screen content image to the client device, and may no longer send the original video stream data to the client device.

[0095] In step 806, the client may display the translated video frame.

[0096] According to an exemplary embodiment of the present disclosure, when the first terminal does not have translation capability and the second terminal has translation capability, the target multimedia data and a translation instruction may be sent to the second terminal, wherein the translation instruction may be used to instruct the second terminal to translate the target multimedia data.

[0097] For audio content: If the host does not have ASR capability and only the client has ASR capability, the host can send the audio content on the host device to the client device through the VCC App for ASR and translation, and then obtain the original text content corresponding to the audio content and the text content in the specified language obtained by translation.

[0098] Fig. 9 It is a schematic diagram showing the flow of audio translation flow on the client side according to an exemplary embodiment of the present disclosure.

[0099] Reference Fig. 9 In step 901, the host can send the audio stream on the host device to the client device through the VCC App.

[0100] In step 902, when the "speech recognition + translation" function is activated on the Client device, the Client device extracts the audio stream output by the speaker of the Host end and the audio stream collected by the microphone of the Host end in the audio stream.

[0101] In step 903, the client device may perform ASR on the audio stream output by the speaker and the audio stream collected by the microphone to obtain the original text content.

[0102] In step 904, the client device may translate the original text content obtained by voice recognition to obtain text content in a specified language.

[0103] In step 905, the Client device may display the original text content and the translated content corresponding to the audio stream output by the speaker, and the original text content and the translated content corresponding to the audio stream collected by the microphone on the screen.

[0104] For the screen content: If the Host does not support OCR+translation capabilities, and only the Client device supports OCR+translation capabilities, you can call the OCR+translation capabilities of the Client device to perform character recognition and translation on the video frames sent by the Host device.

[0105] Fig.10 It is a schematic diagram showing the flow of screen translation on the client side according to an exemplary embodiment of the present disclosure.

[0106] Reference Fig.10 In step 1001, the host sends the video stream to the client.

[0107] In step 1002, the client may capture the bitmap of the current frame of the received video stream every 2 seconds.

[0108] In step 1003, the client can compare the two bitmaps captured before and after to determine whether the screen content displayed by the host has been updated. If the bitmap has not changed, return to step 1003; if the bitmap has changed, execute step 1004.

[0109] In step 1004, if the bitmap changes, it means that a video frame switch has occurred, that is, the host screen currently switches to display a new video frame. At this time, the client can trigger the OCR instruction to extract the character content to be translated from the current video frame.

[0110] In step 1005, the client may translate the character content extracted from the video frame into character content in a specified language.

[0111] In step 1006, the client can use open Cv to erase the original character content on the video frame, and can use the character content of the specified language obtained by translation to fill the erased position of the video frame to obtain a translated video frame.

[0112] In step 1007, the client may display the translated video frame.

[0113] According to an exemplary embodiment of the present disclosure, when neither the first terminal nor the second terminal has translation capability, the target multimedia data and the translation instruction can be sent to a third-party device with translation capability, wherein the translation instruction can be used to instruct the third-party device to translate the target multimedia data, and the obtained translation content can be sent to the second terminal, and the third-party device can be, but is not limited to, a server. Exemplarily, the third-party device can send the translated content to the second terminal; or, the third-party device can first send the translated content to the first terminal, and then the first terminal can send the translated content to the second terminal, and the present disclosure does not make specific restrictions on this.

[0114] In this way, when neither the first terminal nor the second terminal has translation capabilities, a third-party device with translation capabilities can be enabled to perform translation flow, thereby avoiding the phenomenon that communication is forced to be interrupted due to the fact that neither the first terminal nor the second terminal has translation capabilities. That is, smooth communication can be guaranteed and the efficiency of communication can be improved.

[0115] Fig.11 is a schematic diagram showing the effect of transferring AI translation to a tablet according to an exemplary embodiment of the present disclosure. Fig.11 If the Client device is a mobile electronic device such as a tablet, since the screen of such mobile electronic devices is small, the entire screen of the Client device can be set to display the same proportion of screen content as the Host device, and the translated screen content can replace the original text content and be displayed at the corresponding position of the video frame; the results of audio recognition and translation can be displayed in a floating window. In addition, the original language used and the specified target language for translation can also be displayed on the Client device. For example, the original language used can be Chinese, and the specified target language for translation can be Korean, etc.

[0116] In this way, when the screen of the Client device is small, the audio translation result can be controlled to be displayed in a floating window above the translation result of the screen content, so as to achieve reasonable utilization of the screen space of the Client device and ensure the comprehensiveness and clarity of the information display.

[0117] Fig.12 is a schematic diagram showing the effect of streaming AI translation to a TV according to an exemplary embodiment of the present disclosure. Fig.12If the Client device is an electronic device such as a TV, since the screen of such electronic devices is large, the screen of the Client device can be divided into two areas, and one of the areas can be used to display the translated screen content with the screen ratio of the Host device. Furthermore, the translated screen content can replace the original text content and be displayed at the corresponding position of the video frame; the other area can be used to display the results of audio recognition and translation.

[0118] In this way, when the screen of the Client device is large, the translation flow results of the audio and the translation flow results of the screen content can be displayed separately in a split screen manner, which can make full use of the screen space of the Client device and ensure the comprehensiveness and rationality of the information display.

[0119] According to an exemplary embodiment of the present disclosure, the translated content can be projected to a second terminal. Specifically, the Client device may not have AI translation capabilities or the above-mentioned VCC App is not installed. In this case, the target multimedia data needs to be translated using the AI ​​translation capabilities of the Host, and the Host can use the projection protocol to transmit content with the Client device. Exemplarily, the projection protocol can be, but is not limited to, the miracast protocol. Fig.13 is a schematic diagram showing that only the Host end has AI translation capability according to an exemplary embodiment of the present disclosure. Fig.13 The translation of audio content and screen content is all completed on the Host side, and the Host can project the translated content to the Client side for display through the Miracast protocol. Fig.14 2 is a schematic diagram showing the effect of projecting the translated content to the Client device using the projection protocol according to an exemplary embodiment of the present disclosure. Fig.14 , Smart View function can support miracast protocol, so the translated content of Host can be projected to Client through Smart View app for display. At this time, the client can simultaneously display the original screen content, the translated content obtained by translating the original screen content, the original subtitles obtained by voice recognition of the audio content, and the translated subtitles of the original subtitles. In addition, users can also modify the layout of the translated content of audio and the translated content of video frames on the screen of Client according to actual needs.

[0120] According to an exemplary embodiment of the present disclosure, original content may be displayed on a first terminal, wherein the original content may include screen content and / or subtitle content, and translated content corresponding to the original content is used for display on a second terminal. Fig.15is a schematic diagram showing a translated screen content displayed on a second terminal according to an exemplary embodiment of the present disclosure. Fig.15 , the second terminal displays the Korean screen content obtained by translating the original Chinese screen content.

[0121] According to an exemplary embodiment of the present disclosure, the original content may be displayed on the first terminal, wherein the original content may include screen content and / or subtitle content. Furthermore, the original content may be sent to the second terminal, wherein the original content and the translated content corresponding to the original content are used for display on the second terminal. Fig.16 2 is a schematic diagram illustrating displaying translated Korean screen content, original Chinese subtitle content, and translated Korean subtitle content on a second terminal according to an exemplary embodiment of the present disclosure.

[0122] According to an exemplary embodiment of the present disclosure, in the case where the original content includes subtitle content, the subtitle content may not be displayed on the first terminal, and the subtitle content and translation content corresponding to the subtitle content may be used for display on the second terminal. Fig.17 2 is a schematic diagram showing displaying original subtitle content and translated subtitle content on a second terminal according to an exemplary embodiment of the present disclosure. Fig.17 , the second terminal displays the original Chinese subtitle content and the Korean subtitle content obtained by translating the original Chinese subtitle content.

[0123] According to an exemplary embodiment of the present disclosure, original content and translated content corresponding to the original content may be displayed on a first terminal, wherein the original content may include screen content and / or subtitle content. Furthermore, the original content may be sent to a second terminal, wherein the original content and translated content corresponding to the original content are used for display on the second terminal.

[0124] It should be noted that when a participant wants to end the translation flow during an online meeting, he or she can click the End button, which will disconnect the connection between the Host device and the Client device, thereby stopping the translation flow.

[0125] Fig.18 The present invention is a flow chart showing another multimedia data transmission method according to an exemplary embodiment of the present disclosure. The method is applied to a second terminal, and the second terminal can communicate multimedia data with the first terminal.

[0126] Reference Fig.18In step 1801, when the first terminal has translation capability, translation content for target multimedia data in the first terminal can be received from the first terminal, wherein the translation content can be obtained by the first terminal translating the target multimedia data in response to receiving an instruction to start multimedia data translation.

[0127] According to an exemplary embodiment of the present disclosure, the target multimedia data may include the screen content of the first terminal and / or the subtitle content obtained by performing voice recognition on the audio content of the first terminal. The audio content may be the speech content of the participants during the communication process, or the audio file referenced by the shared video material; the screen content may be the shared document content, video file, PPT content, etc.

[0128] According to an exemplary embodiment of the present disclosure, a translation capability query request may also be received from the first terminal. Then, in response to the translation capability query request, information about the translation capability of the second terminal may be sent to the first terminal.

[0129] According to an exemplary embodiment of the present disclosure, when the first terminal does not have translation capability and the second terminal has translation capability, target multimedia data and a translation instruction can be received from the first terminal. Then, in response to the received translation instruction, the target multimedia data can be translated to obtain translation content.

[0130] According to an exemplary embodiment of the present disclosure, when neither the first terminal nor the second terminal has translation capability, translation content for target multimedia data can be received, wherein the translation content can be obtained by translating the target multimedia data by a third-party device with translation capability. The third-party device can be, but is not limited to, a server. Exemplarily, the third-party device can send the translated content to the second terminal; or, the third-party device can first send the translated content to the first terminal, and then the first terminal can send the translated content to the second terminal, and the present disclosure does not make specific restrictions on this.

[0131] In this way, when neither the first terminal nor the second terminal has translation capabilities, a third-party device with translation capabilities can be enabled to perform translation flow, thereby avoiding the phenomenon that communication is forced to be interrupted due to the fact that neither the first terminal nor the second terminal has translation capabilities. That is, smooth communication can be guaranteed and the efficiency of communication can be improved.

[0132] According to an exemplary embodiment of the present disclosure, the translation content projected from the first terminal can be received. Specifically, the Client device may not have AI translation capabilities or the aforementioned VCC App is not installed. In this case, it is necessary to use the AI ​​translation capabilities of the Host to translate the target multimedia data. In addition, the Host can use the projection protocol to transmit content with the Client device. Exemplarily, the projection protocol can be, but is not limited to, the miracast protocol.

[0133] In step 1802, the translated content may be displayed.

[0134] According to an exemplary embodiment of the present disclosure, the translated content corresponding to the original content may be displayed on the second terminal, wherein the original content may include screen content and / or subtitle content. Furthermore, the original content may also be displayed on the first terminal.

[0135] According to an exemplary embodiment of the present disclosure, original content can also be received from the first terminal, wherein the original content can include screen content and / or subtitle content, and the original content can also be displayed on the first terminal. In addition, the original content and the translation content corresponding to the original content can also be displayed on the second terminal.

[0136] According to an exemplary embodiment of the present disclosure, in a case where original content includes subtitle content, the subtitle content and translated content corresponding to the subtitle content may be displayed on the second terminal, wherein the subtitle content may not be displayed on the first terminal.

[0137] According to an exemplary embodiment of the present disclosure, the original content and the translation content corresponding to the original content are used to be displayed on the first terminal, wherein the original content may include screen content and / or subtitle content. In addition, the original content may be received from the first terminal, and the original content and the translation content corresponding to the original content may be displayed on the second terminal.

[0138] Fig.19 1 is a block diagram illustrating a multimedia data transmission apparatus 1900 according to an exemplary embodiment of the present disclosure.

[0139] Reference Fig.19 The multimedia data transmission device 1900 may include a multimedia data acquisition module 1901 and a translation module 1902.

[0140] In response to receiving the instruction to start multimedia data translation, the multimedia data acquisition module 1901 can acquire the target multimedia data in the first terminal.

[0141] According to an exemplary embodiment of the present disclosure, the target multimedia data may include the screen content of the first terminal and / or the subtitle content obtained by performing voice recognition on the audio content of the first terminal. The audio content may be the speech content of the participants during the communication process, or the audio file referenced by the shared video material; the screen content may be the shared document content, video file, PPT content, etc.

[0142] When the first terminal has translation capability, the translation module 1902 can translate the target multimedia data, obtain the translation content, and send the translation content to the second terminal for display on the second terminal. That is, the above-mentioned VCC App can perform translation transfer according to the AI ​​capabilities of the host and the client, including the translation transfer of audio content and / or the translation transfer of screen content.

[0143] According to an exemplary embodiment of the present disclosure, the target multimedia data may be audio content. Speech recognition may be performed on the audio content. Exemplarily, ASR technology may be used to perform speech recognition on the audio content to obtain subtitle content corresponding to the audio content, i.e., text content. Then, the text content may be translated into text content in the target language.

[0144] In this way, by translating the audio content generated during the meeting into text content in a specified language, it is convenient for users on the client side who do not speak the language to timely understand the specific content of the communication between the participating users, thereby ensuring the smooth progress of the meeting and improving the efficiency of communication.

[0145] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 1900 may further include a query module and a main device determination module.

[0146] The query module can query whether the first terminal and / or the second terminal has the translation capability. Then, the main device determination module can determine the main device for translating the target multimedia data according to the query result.

[0147] According to an exemplary embodiment of the present disclosure, the query module may send a translation capability query request to the second terminal. Then, the query module may receive information about the translation capability of the second terminal from the second terminal.

[0148] According to an exemplary embodiment of the present disclosure, when the first terminal has a translation capability, the first terminal may translate the target multimedia data to obtain translation content.

[0149] For the audio content, you can determine whether to use the Host device or the Client device for audio content translation and transfer based on whether the Host device and the Client device support ASR capabilities. Specifically, if both the Host device and the Client device have ASR capabilities, the Host device can be used for ASR and translation first; if only the Host device has ASR capabilities and the Client device does not have ASR capabilities, the Host device can be used for ASR and translation.

[0150] For the screen content, before translating the screen content, the host can send a command to the client to inquire whether the client has the "OCR+translation" capability. Then, based on the actual situation of whether the host and client devices support "OCR+translation", it can be determined whether to use the host device or the client device for screen translation. Specifically, if both the host and client devices support OCR and translation capabilities; or if only the host has OCR and translation capabilities, but the client does not, the "OCR+translation" function of the host device can be called to perform character recognition and translation on the screen content currently displayed on the Host device, and the translated content can be transmitted to the Client device for display through the VCC App.

[0151] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 1900 may further include a first multimedia data sending module.

[0152] When the first terminal does not have translation capability but the second terminal has translation capability, the first multimedia data sending module may send the target multimedia data and a translation instruction to the second terminal, wherein the translation instruction may be used to instruct the second terminal to translate the target multimedia data.

[0153] For audio content: If the host does not have ASR capability and only the client has ASR capability, the host can send the audio content on the host device to the client device through the VCC App for ASR and translation, and then obtain the original text content corresponding to the audio content and the text content in the specified language obtained by translation.

[0154] For the screen content: If the Host does not support OCR+translation capabilities, and only the Client device supports OCR+translation capabilities, you can call the OCR+translation capabilities of the Client device to perform character recognition and translation on the video frames sent by the Host device.

[0155] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 1900 may further include a second multimedia data sending module.

[0156] When neither the first terminal nor the second terminal has the translation capability, the second multimedia data sending module can send the target multimedia data and the translation instruction to a third-party device with the translation capability, wherein the translation instruction can be used to instruct the third-party device to translate the target multimedia data, and the obtained translation content can be sent to the second terminal, and the third-party device can be, but is not limited to, a server. Exemplarily, the third-party device can send the translated content to the second terminal; or the third-party device can first send the translated content to the first terminal, and then the first terminal can send the translated content to the second terminal, and the present disclosure does not make specific restrictions on this.

[0157] In this way, when neither the first terminal nor the second terminal has translation capabilities, a third-party device with translation capabilities can be enabled to perform translation flow, thereby avoiding the phenomenon that communication is forced to be interrupted due to the fact that neither the first terminal nor the second terminal has translation capabilities. That is, smooth communication can be guaranteed and the efficiency of communication can be improved.

[0158] According to an exemplary embodiment of the present disclosure, the translation module 1902 can project the translated content to the second terminal. Specifically, the Client device may not have AI translation capabilities or the VCC App is not installed. In this case, the host's AI translation capabilities need to be used to translate the target multimedia data, and the host can use the projection protocol to transmit content with the Client device. Exemplarily, the projection protocol can be, but is not limited to, the miracast protocol.

[0159] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 1900 may further include a first original content display module. The first original content display module may display original content on the first terminal, wherein the original content may include screen content and / or subtitle content, and the translation content corresponding to the original content is used to be displayed on the second terminal.

[0160] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 1900 may further include a second original content display module and a first original content sending module.

[0161] The second original content display module can display the original content on the first terminal, wherein the original content can include screen content and / or subtitle content. Furthermore, the first original content sending module can send the original content to the second terminal, wherein the original content and the translation content corresponding to the original content are used for display on the second terminal.

[0162] According to an exemplary embodiment of the present disclosure, in the case where the original content includes subtitle content, the subtitle content may not be displayed on the first terminal, and the subtitle content and translation content corresponding to the subtitle content may be used for display on the second terminal.

[0163] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 1900 may further include an original content and translation content display module and a second original content sending module.

[0164] The original content and translation content display module can display the original content and the translation content corresponding to the original content on the first terminal, wherein the original content can include screen content and / or subtitle content. In addition, the second original content sending module can send the original content to the second terminal, wherein the original content and the translation content corresponding to the original content are used for display on the second terminal.

[0165] Fig. 20 2 is a block diagram illustrating another multimedia data transmission apparatus 2000 according to an exemplary embodiment of the present disclosure.

[0166] Reference Fig. 20 The multimedia data transmission device 2000 may include a first translation content receiving module 2001 and a translation content display module 2002 .

[0167] The first translation content receiving module 2001 can receive translation content for target multimedia data in the first terminal from the first terminal when the first terminal has translation capability, wherein the translation content can be obtained by the first terminal translating the target multimedia data in response to receiving an instruction to start multimedia data translation.

[0168] According to an exemplary embodiment of the present disclosure, the target multimedia data may include the screen content of the first terminal and / or the subtitle content obtained by performing voice recognition on the audio content of the first terminal. The audio content may be the speech content of the participants during the communication process, or the audio file referenced by the shared video material; the screen content may be the shared document content, video file, PPT content, etc.

[0169] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 2000 may further include a query request receiving module and a translation capability information sending module.

[0170] The query request receiving module may also receive a translation capability query request from the first terminal. Then, the translation capability information sending module may send information about the translation capability of the second terminal to the first terminal in response to the translation capability query request.

[0171] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 2000 may further include a multimedia data receiving module and a translation module.

[0172] The multimedia data receiving module can receive target multimedia data and a translation instruction from the first terminal when the first terminal does not have a translation capability and the second terminal has a translation capability. Then, the translation module can translate the target multimedia data in response to the received translation instruction to obtain translation content.

[0173] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 2000 may further include a second translation content receiving module.

[0174] The second translation content receiving module can receive translation content for target multimedia data when neither the first terminal nor the second terminal has translation capability, wherein the translation content can be obtained by translating the target multimedia data by a third-party device with translation capability. The third-party device can be, but is not limited to, a server. Exemplarily, the third-party device can send the translated content to the second terminal; or, the third-party device can first send the translated content to the first terminal, and then the first terminal can send the translated content to the second terminal, and the present disclosure does not impose specific restrictions on this.

[0175] In this way, when neither the first terminal nor the second terminal has translation capabilities, a third-party device with translation capabilities can be enabled to perform translation flow, thereby avoiding the phenomenon that communication is forced to be interrupted due to the fact that neither the first terminal nor the second terminal has translation capabilities. That is, smooth communication can be guaranteed and the efficiency of communication can be improved.

[0176] According to an exemplary embodiment of the present disclosure, the first translation content receiving module 2001 can receive the translation content projected from the first terminal. Specifically, the Client device may not have AI translation capabilities or the aforementioned VCCApp is not installed. At this time, it is necessary to use the AI ​​translation capabilities of the Host to translate the target multimedia data. In addition, the Host can use the screen projection protocol to transmit content with the Client device. Exemplarily, the screen projection protocol can be but is not limited to: miracast protocol.

[0177] The translation content display module 2002 can display the translation content.

[0178] According to an exemplary embodiment of the present disclosure, the translation content display module 2002 may display the translation content corresponding to the original content, wherein the original content may include screen content and / or subtitle content. Furthermore, the original content may also be displayed on the first terminal.

[0179] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 2000 may further include a first original content receiving module.

[0180] The first original content receiving module can also receive original content from the first terminal, wherein the original content can include screen content and / or subtitle content, and the original content can also be displayed on the first terminal. In addition, the above-mentioned translation content display module 2002 can also display the original content and the translation content corresponding to the original content on the second terminal.

[0181] According to an exemplary embodiment of the present disclosure, the above-mentioned translation content display module 2002 can display the subtitle content and the translation content corresponding to the subtitle content on the second terminal when the original content includes subtitle content, wherein the subtitle content may not be displayed on the first terminal.

[0182] According to an exemplary embodiment of the present disclosure, the multimedia data transmission device 2000 may further include a second original content receiving module.

[0183] The original content and the translation content corresponding to the original content are used to be displayed on the first terminal, wherein the original content may include screen content and / or subtitle content. In addition, the second original content receiving module may also receive the original content from the first terminal, and then the above-mentioned translation content display module 2002 may display the original content and the translation content corresponding to the original content on the second terminal.

[0184] Fig.21 is a block diagram illustrating an electronic device 2100 according to an exemplary embodiment of the present disclosure.

[0185] Reference Fig.21 The electronic device 2100 includes at least one memory 2101 and at least one processor 2102, wherein the at least one memory 2101 stores instructions, and when the instructions are executed by the at least one processor 2102, a method for transmitting multimedia data according to an exemplary embodiment of the present disclosure is executed.

[0186] As an example, the electronic device 2100 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 2100 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 2100 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0187] In the electronic device 2100, the processor 2102 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0188] The processor 2102 may execute instructions or codes stored in the memory 2101, wherein the memory 2101 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0189] The memory 2101 may be integrated with the processor 2102, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 2101 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 2101 and the processor 2102 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 2102 can read files stored in the memory.

[0190] In addition, the electronic device 2100 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 2100 may be connected to each other via a bus and / or a network.

[0191] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned method for transmitting multimedia data. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0192] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, and when the computer program is executed by a processor, the method for transmitting multimedia data according to the present disclosure is implemented.

[0193] According to the multimedia data transmission method, device, electronic device, storage medium and computer program product disclosed in the present invention, in a multi-user communication scenario, if the multiple users do not speak the same language, the AI ​​translation function can be enabled, so that while communicating through the host device, the corresponding translation content can be displayed on the client device that needs translation. In this way, users who need translation can understand the specific content of the communication in a timely manner, and then can ensure the smooth progress of the communication process, thereby improving the efficiency of communication.

[0194] According to an exemplary embodiment of the present disclosure, by translating the audio content generated during the meeting into text content in a specified language, it is convenient for users on the client side who do not understand the language to timely understand the specific content of the communication between each participating user, thereby ensuring the smooth progress of the meeting and improving the efficiency of communication.

[0195] According to an exemplary embodiment of the present disclosure, in the case that neither the first terminal nor the second terminal has translation capability, a third-party device with translation capability can be enabled to perform translation flow, thereby avoiding the phenomenon that communication is forced to be interrupted due to the fact that neither the first terminal nor the second terminal has translation capability, that is, the smooth conduct of communication can be guaranteed and the efficiency of communication can be improved.

[0196] According to an exemplary embodiment of the present disclosure, when the screen of the Client device is small, the audio translation result can be controlled to be displayed in a floating window above the translation result of the screen content, thereby realizing reasonable utilization of the screen space of the Client device and ensuring the comprehensiveness and clarity of the information display.

[0197] According to an exemplary embodiment of the present disclosure, when the screen of the Client device is relatively large, the translation flow results of the audio and the translation flow results of the screen content can be displayed separately in a split-screen manner, thereby fully utilizing the screen space of the Client device and ensuring the comprehensiveness and rationality of the information display.

[0198] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0199] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for transmitting multimedia data, applied to a first terminal, wherein the first terminal communicates multimedia data with a second terminal, characterized in that: include: In response to receiving an instruction to start multimedia data translation, acquiring target multimedia data in the first terminal; In the case where the first terminal has a translation capability, the target multimedia data is translated to obtain translation content, and the translation content is sent to the second terminal for display on the second terminal.

2. The transmission method according to claim 1, characterized in that: Also includes: In the case that the first terminal does not have translation capability but the second terminal has translation capability, the target multimedia data and a translation instruction are sent to the second terminal, wherein the translation instruction is used to instruct the second terminal to translate the target multimedia data.

3. The transmission method according to claim 1, characterized in that: Also includes: When neither the first terminal nor the second terminal has translation capability, the target multimedia data and the translation instruction are sent to a third-party device with translation capability, wherein the translation instruction is used to instruct the third-party device to translate the target multimedia data, and the obtained translation content is sent to the second terminal.

4. The transmission method according to claim 1, characterized in that: Also includes: querying whether the first terminal and / or the second terminal has translation capability; According to the query result, a main device for translating the target multimedia data is determined.

5. The transmission method according to claim 4, characterized in that: The querying whether the second terminal has the translation capability includes: Sending a translation capability query request to the second terminal; Information about a translation capability of the second terminal is received from the second terminal.

6. The transmission method according to claim 1, characterized in that: The sending the translated content to the second terminal includes: Projecting the translated content to the second terminal.

7. The transmission method according to claim 1, characterized in that: The target multimedia data includes screen content of the first terminal and / or subtitle content obtained by performing speech recognition on audio content of the first terminal.

8. The transmission method according to claim 7, characterized in that: Also includes: Displaying original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; The translated content corresponding to the original content is used for display on the second terminal.

9. The transmission method according to claim 7, characterized in that: Also includes: Displaying original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; The original content is sent to the second terminal, wherein the original content and the translated content corresponding to the original content are used for display on the second terminal.

10. The transmission method according to claim 7, characterized in that: Also includes: Displaying original content and translated content corresponding to the original content on the first terminal, wherein the original content includes the screen content and / or the subtitle content; The original content is sent to the second terminal, wherein the original content and the translated content corresponding to the original content are used for display on the second terminal.

11. The transmission method according to claim 9, characterized in that: In the case that the original content includes the subtitle content, the subtitle content is not displayed on the first terminal, and the subtitle content and the translation content corresponding to the subtitle content are used to be displayed on the second terminal.

12. A method for transmitting multimedia data, applied to a second terminal, wherein the second terminal communicates multimedia data with a first terminal, characterized in that: include: In the case where the first terminal has a translation capability, receiving translation content for target multimedia data in the first terminal from the first terminal, wherein the translation content is obtained by the first terminal translating the target multimedia data in response to receiving an instruction to start multimedia data translation; Display the translation.

13. The transmission method according to claim 12, characterized in that: Also includes: In a case where the first terminal does not have a translation capability and the second terminal has a translation capability, receiving the target multimedia data and a translation instruction from the first terminal; In response to the translation instruction, the target multimedia data is translated to obtain translation content.

14. The transmission method according to claim 12, characterized in that: Also includes: In the case that neither the first terminal nor the second terminal has a translation capability, translation content for the target multimedia data is received, wherein the translation content is obtained by translating the target multimedia data by a third-party device with a translation capability.

15. The transmission method according to claim 12, characterized in that: Also includes: receiving a translation capability query request from the first terminal; In response to the translation capability query request, information about the translation capability of the second terminal is sent to the first terminal.

16. The transmission method according to claim 12, characterized in that: The receiving, from the first terminal, the translation content for the target multimedia data in the first terminal includes: Receive the translation content projected from the first terminal.

17. The transmission method according to claim 12, characterized in that: The target multimedia data includes screen content of the first terminal and / or subtitle content obtained by performing speech recognition on audio content of the first terminal.

18. The transmission method according to claim 17, characterized in that: The displaying of the translated content includes: Displaying translated content corresponding to original content on the second terminal, wherein the original content includes the screen content and / or the subtitle content; The original content is also displayed on the first terminal.

19. The transmission method according to claim 17, characterized in that: Also includes: Receiving original content from the first terminal, wherein the original content includes the screen content and / or the subtitle content, and the original content is also displayed on the first terminal; The display of the translation content includes: The original content and the translated content corresponding to the original content are displayed on the second terminal.

20. The transmission method according to claim 17, characterized in that: The original content and the translated content corresponding to the original content are used to be displayed on the first terminal, wherein the original content includes the screen content and / or the subtitle content; Also includes: receiving the original content from the first terminal; The displaying of the translated content includes: The original content and the translated content corresponding to the original content are displayed on the second terminal.

21. The transmission method according to claim 19, characterized in that: In a case where the original content includes the subtitle content, displaying the original content and the translation content corresponding to the original content on the second terminal includes: Displaying the subtitle content and translation content corresponding to the subtitle content on the second terminal; The subtitle content is not displayed on the first terminal.

22. A multimedia data transmission device, applied to a first terminal, and the first terminal communicates multimedia data with a second terminal, characterized in that: include: a multimedia data acquisition module, configured to acquire target multimedia data in the first terminal in response to receiving an instruction to start multimedia data translation; The translation module is configured to translate the target multimedia data to obtain translation content when the first terminal has translation capability, and send the translation content to the second terminal for display on the second terminal.

23. A multimedia data transmission device, applied to a second terminal, wherein the second terminal communicates multimedia data with a first terminal, characterized in that: include: A first translation content receiving module is configured to receive translation content for target multimedia data in the first terminal from the first terminal if the first terminal has a translation capability, wherein the translation content is obtained by the first terminal translating the target multimedia data in response to receiving an instruction to start multimedia data translation; The translation content display module is configured to display the translation content.

24. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instruction to implement the multimedia data transmission method according to any one of claims 1 to 11, or to implement the multimedia data transmission method according to any one of claims 12 to 21.

25. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the multimedia data transmission method as described in any one of claims 1 to 11, or to execute the multimedia data transmission method as described in any one of claims 12 to 21.

26. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for transmitting multimedia data according to any one of claims 1 to 11 is implemented, or the method for transmitting multimedia data according to any one of claims 12 to 21 is implemented.

Images