Cross-language call method, device and storage medium

Abstract

A method includes: determining description information of the plurality of call devices, wherein the description information comprises language category adopted by the plurality of call devices and translation capability of the call devices; determining a language category set of the plurality of the call devices; obtaining first text information, wherein the first text information is converted based on a first audio file obtained from a call among the plurality of call devices; obtaining second text information, wherein the second text information is translated by at least one of the plurality of call devices based on the translation capability to convert the first text information into at least one language of the language category set; sending the second text information to at least one target call device among the plurality of call devices, whereby the target call device converts the second text information into a second audio file.

Description

CROSS-LANGUAGE CALL METHOD, DEVICE AND STORAGE MEDIUM

[0001] The present disclosure relates to the technical field of communications, and more particularly, to a cross-language call method, a device and a storage medium.

[0002] With the popularity of audio and video calls for meetings and other scenarios, audio and video communication based on mobile terminals or television devices has become more and more common. When multiple people have a cross-multilingual meeting, manual translation is required usually due to differences in language types.

[0003] Currently, large language models may be deployed at device sides to achieve automatic translation. However, some devices do not support the deployment of large models and cannot enjoy automatic translation services.

[0004] The present disclosure provides a cross-language call method, apparatus and system, a device, a storage medium, and a product, to achieve convenient cross-language calls.

[0005] The technical solutions of embodiments of the present disclosure are as follows.

[0006] In one embodiment, a cross-language call method is performed by a plurality of call devices. In one embodiment, the method may include determining description information of the plurality of call devices, wherein the description information comprises language category adopted by the plurality of call devices and translation capability of the call devices;

[0007] In one embodiment, the method may include determining a language category set of the plurality of call devices based on the language category adopted by the plurality of call devices;

[0008] In one embodiment, the method may include obtaining first text information, wherein the first text information is converted based on a first audio file obtained from a call among the plurality of call devices;

[0009] In one embodiment, the method may include obtaining second text information, wherein the second text information is translated by at least one of the plurality of call devices based on the translation capability to convert the first text information into at least one language of the language category set;

[0010] In one embodiment, the method may include sending the second text information to at least one target call device among the plurality of call devices, wherein the language category adopted by the at least one target call device is the same as the language category of the second text information;

[0011] In one embodiment, the method may include obtaining a second audio file, wherein the second audio file is generated by the target call device converting the second text information into an audio file of the language category adopted by the target call device.

[0012] In one embodiment, an electronic device may include at least one processor including processing circuitry, memory storing instructions that, when executed by the at least one processor individually or collectively.

[0013] In one embodiment, an electronic device may include memory storing instructions.

[0014] In one embodiment, the at least one processor may cause the electronic device to determine description information of the plurality of call devices, wherein the description information comprises language category adopted by the plurality of call devices and translation capability of the plurality of call devices.

[0015] In one embodiment, the at least one processor may cause the electronic device to determine a language category set of the plurality of call devices based on the language category adopted by the plurality of call devices.

[0016] In one embodiment, the at least one processor may cause the electronic device to obtain first text information, wherein the first text information is converted based on a first audio file obtained from a call among the plurality of call devices.

[0017] In one embodiment, the at least one processor may cause the electronic device to obtain second text information, wherein the second text information is translated by at least one of based on the translation capability to convert the first text information into at least one language category.

[0018] In one embodiment, the at least one processor may cause the electronic device to send the second text information to at least one target call device among the plurality of call devices, wherein the language category adopted by the at least one target call device is the same as the language category of the second text information.

[0019] The memory has an application program executable by the processor stored therein for causing the processor to perform the cross-language call method as described in any one of the above.

[0020] A computer-readable storage medium has computer-readable instructions stored therein for performing the cross-language call method as described in any one of the above.

[0021] It can be seen from the above technical solutions that in embodiments of the present disclosure, description information of each call device in a call device group is determined, where the description information characterizes language categories adopted by the call device and translation capabilities of the call device; a language category set of the call device group is determined; first text information is determined, where the first text information is obtained based on converting a first audio file during an audio and video call of the call device group; based on the translation capability of each call device, the first text information is distributed to a translative call device belonging to the call device group and supporting translation of the first text information into the language categories in the language category set, so as to translate the first text information into second text information; and the second text information is sent to a target call device in the call device group, whereby the target call device converts the second text information into a second audio file. It can be seen therefrom that according to the embodiments of the present disclosure, translation tasks are allocated reasonably based on device capabilities. Even if a device does not support a translation capability, cross-language calls can be achieved based on translation results provided by other devices, so as to enjoy translation services.

[0022] Furthermore, according to the embodiments of the present disclosure, a concept of groups is established for devices having the translation capability, which cooperate under the unified administration of an administrative call device, whereby device resources can be utilized reasonably. In addition, the embodiments of the present disclosure are applicable to one-to-many translation scenarios. A receiving end according to the embodiments of the present disclosure can only receive translated text and convert the text into audio, thereby avoiding the problem of multi-audio mixing. According to the embodiments of the present disclosure, text formatted data with less storage is transmitted, thereby effectively reducing network delay. According to the embodiments of the present disclosure, a situation of disordered multi-person translation is avoided. According to the embodiments of the present disclosure, device usage can be dynamically monitored, and the administrative call device can be dynamically adjusted, thereby improving user experience.

[0023] Fig. 1 is an exemplary flowchart of a cross-language call method according to an embodiment of the present disclosure.

[0024] Fig. 2A is an exemplary schematic diagram of reporting description information according to an embodiment of the present disclosure.

[0025] Fig. 2B is an exemplary transmission diagram of a signaling stream and a media stream according to an embodiment of the present disclosure.

[0026] Fig. 2C is an exemplary schematic diagram of a format structure of a session description protocol message according to an embodiment of the present disclosure.

[0027] Fig. 3 is an exemplary schematic diagram of a text sorting process according to an embodiment of the present disclosure.

[0028] Fig. 4 is a block diagram of an administrative call device according to an embodiment of the present disclosure.

[0029] Fig. 5 is an exemplary schematic diagram of a cross-language call process according to an embodiment of the present disclosure.

[0030] Fig. 6 is an exemplary structural diagram of a cross-language call system according to an embodiment of the present disclosure.

[0031] Fig. 7A is an exemplary schematic diagram of a one-to-one audio and video call according to an embodiment of the present disclosure.

[0032] Fig. 7B is an exemplary schematic diagram of a one-to-many audio and video call according to an embodiment of the present disclosure.

[0033] Fig. 7C is an exemplary schematic diagram of a one-to-many audio and video call and requiring replacement of an administrative call device in accordance with an embodiment of the present disclosure.

[0034] Fig. 7D is an exemplary schematic diagram of dynamically monitoring a device state change according to an embodiment of the present disclosure.

[0035] Fig. 8 is an exemplary schematic diagram of a cross-language call apparatus according to an embodiment of the present disclosure.

[0036] Fig. 9 is an exemplary structural diagram of an electronic device having a memory-processor architecture according to the present disclosure.

[0037] In order that the objects, technical solutions, and advantages of the present disclosure will become more apparent, the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0038] For simplicity and clarity of description, the solutions of the present disclosure are set forth below by describing a representative embodiment. Numerous details in an embodiment are set forth only to provide an understanding of the solutions of the present disclosure. However, it will be apparent that the technical solutions of the present disclosure may be implemented without being limited to these details. To avoid unnecessarily obscuring the solutions of the present disclosure, an embodiment have not been described in detail, but rather have been provided with a framework. Hereinafter, "including" means "including but not limited to", and "according to" means "at least according to, but not limited to only according to". Because of Chinese language conventions, when the number of one component is not specified below, it means that there may be one or more components, or it may be understood that there is at least one component.

[0039] In recent years, with the popularity of meetings including audio and video calls and other scenarios, cross-language audio and video communication based on mobile terminals (such as mobile phones) or television devices has become commonplace. In a multi-person and cross-multilingual meeting scenario, multiple languages often need to be translated because of language communication issues. Large models for translation may be deployed at a device side. However, in practical use, the following problems are often encountered: (1) Some devices do not support translation or large model processing capabilities and cannot enjoy translation services. (2) Translation is performed only at one device (e.g., transmitter), and audio obtained by conversion is transmitted to the other device (e.g., receiver). However, the capabilities of the translator are limited. If the resources are tight, the translator may fail to translate properly. (3) When multiple people join a video meeting, the requirements for multilingual translation are usually not met. (4) In a noisy environment around a speaker, information transmitted to the opposite side is disordered. (5) During the translation, audio information is transmitted with a long delay time. (6) When the network is used for translation, the privacy of users is difficult to be guaranteed.

[0040] For example: at present, some smart terminals have supported real-time call translation, but a receiving end needs to receive an original audio and translated audio of a user, and there is a large delay between the two audios. If the transmitter then speaks a second time, the receiver will hear both a first translated audio and an original audio of the second conversation. In addition, current smart terminals typically support call translation of only two participants and do not support call translation of multiple participants using multiple languages.

[0041] An embodiment of the present disclosure provide a method for improving real-time call translation based on dynamic decisions. According to processing capabilities of different user equipment, based on the control of an administrative call device, various devices dynamically negotiate to jointly complete translation tasks, and the change of device capabilities may also be monitored. If the administrative call device is not suitable, the administrative call device is dynamically replaced, thereby breaking through the way that only one-to-one calls can be made in the related technology. The method is suitable for one-to-many scenarios, and devices with no capability may also enjoy translation services. Furthermore, in a multi-person speaking scenario, a speech of only a key person may be translated, and the transmission content is text. Therefore, compared with audio transmission in the related technology, the transmission has a shorter delay. At the same time, the capability changes of a transmitting end and a receiving end may be monitored in real time so as to adjust task allocation in real time, thereby improving user experience.

[0042] The above disclosure describes in detail technical defects existing in the related art, the causes leading to the technical defects, and the process of thinking and analysis to overcome the technical defects. In fact, the recognition of the above technical defects is not common knowledge in the art, but novel findings of the applicants in research. In addition, the reason tracing of the technical defects and the process of thinking and analysis to overcome the technical defects are both the stepwise analysis results of the applicants in the actual research process, and are neither common knowledge in the art.

[0043] Fig. 1 is an exemplary flowchart of a cross-language call method according to an embodiment of the present disclosure. The method is performed by an administrative call device in a call device group. For example, a caller during an audio and video call may be determined as a default administrative call device. The administrative call device may be updated based on capabilities of call devices (e.g., basic capabilities and professional capabilities). As shown in Fig. 1, the method includes the following steps:

[0044] Step 101: Determine description information of a plurality of call devices in the call device group , wherein the description information comrpises language category adopted by the plurality of call devices and translation capability of the plurality of call devices.

[0045] The call device group may include multiple call devices, where at least one call device is an administrative call device.

[0046] In one embodiment, step 101 includes: receiving, in a signaling connection during an audio and video call, description information of each call device based on a session description protocol (SDP) message, where the description information is carried in a description field of the SDP message.

[0047] Fig. 2A is an exemplary schematic diagram of reporting description information according to an embodiment of the present disclosure. Fig. 2B is an exemplary transmission diagram of a signaling stream and a media stream according to an embodiment of the present disclosure. Fig. 2C is an exemplary schematic diagram of a format structure of a session description protocol message according to an embodiment of the present disclosure.

[0048] When two or more users establish a connection through the signaling Server, exchange the new defined SDP information in this disclosure, so as to establish the initial communication. As shown in Fig. 2B, media interaction between devices in a device group is implemented based on a Turn server 225, and signaling interaction between the devices is implemented based on a signaling server 220. As shown in Fig. 2A, when device A 210 enters a conversation process of call devices, device A 210 creates an SDP session based on a command: CreateOffer(), and creates a local description of device A 210 based on a command: SetLocalDescription(). For example, the local description of device A 210 may include the following: content (1): information such as a media transmission mode, an encoding format and a network address supported by device A 210; and content (2): a language category adopted by device A 210 and a translation capability of device A 210. Device A 210 carries content (1) in corresponding fields such as "network description", "stream description", "security description" and "Qos and group description" in Fig. 2C, and carries content (2) in a field "adopted language category and translation capability description" in Fig. 2C to form an SDP message of device A 210(A SDP). The SDP message of device A 210(A SDP) is sent to the signaling server 220. The signaling server 220 sends the SDP message of device A 210(A SDP) to device B 230. Device B 230 obtains the local description of device A 210 based on a command: SetRemoteDescription, joins the SDP session based on a command: CreateAnswer(), and creates a local description of device B 230 based on a command: SetLocalDescription(). Similarly, the local description of device B 230 includes the following: content (3): information such as a media transmission mode, an encoding format and a network address supported by device B 230; and content (4): a language category adopted by device B 230 and a translation capability of device B 230. Device B 230 carries content (3) in corresponding fields such as "network description", "stream description", "security description" and "Qos and group description" in Fig. 2C, and carries content (4) in a field "adopted language category and translation capability description" in Fig. 2C to form an SDP message of device B 230(B SDP). The SDP message of device B 230(B SDP) is sent to the signaling server 220.

[0049] When device A 210 is an administrative call device, the signaling server 220 sends the SDP message from device B 230(B SDP) to device A 210. Device A 210 may obtain the language category used by device B 230 and the translation capability of device B 230 by parsing the field "adopted language category and translation capability description" in the SDP message sent by device B 230(for example, device B 230 adopts Chinese, and device B 230 may support translation of Chinese into German, French, and the like). When device B 230 is an administrative call device, device B 230 may obtain the language category used by device A 210 and the translation capability of device A 210 by parsing the field "adopted language category and translation capability description" in the SDP message sent by device A 210(for example, device A 210 adopts English, and device B 230 may support translation of English into German, French, and the like).

[0050] When establishing an audio and video call connection of an audio and video streaming media protocol, WebRTC is taken as an example, when two or more devices establish a connection via the signaling server 220, SDP information as defined in Fig. 2C is interacted so as to establish an initial connection of communication. Once a new device joins the session, the SDP information needs to be exchanged to establish a connection for the new device. Based on this, if the capability of the administrative call device changes, the change may be sensed by the signaling server 220 and a subsequent update process may be triggered. It can be seen that in an embodiment of the present disclosure, an SDP data format is extended, and a capability description field is added to indicate a translation capability of a device and a language category adopted thereby. In this way, an administrative call device may obtain capabilities (including the translation competence and the language category adopted thereby) of the administrative call device and user device participating in the audio and video call through the signaling interaction of a signaling server.

[0051] Step 102: Determine a language category set of the plurality of call devices based on the language category adopted by the plurality of call devices.

[0052] Here, the meaning of the language category set is: all language categories adopted by all call devices in the call device group.

[0053] For example, when the call device group includes device 1, device 2, device 3, device 4 and device 5, device 1 is in Chinese (to be specific, a speech made by a user of device 1 is Chinese), device 2 is in Chinese, device 3 is in English, device 4 is in Japanese, and device 5 is in French. Then the language category set includes: (Chinese, English, Japanese, and French).

[0054] Step 103: Obtain first text information, wherein the first text information is converted based on a first audio obtained from a call among the plurality of call devices.

[0055] For example, the first audio file may be converted to the first text information based on an automatic speech recognition (ASR) technology at the corresponding call device for acquiring the first audio file. The language category of the first text information is the language category adopted by the call device for acquiring the first audio file.

[0056] Step 104: Obtain second text information, wherein the second text information is translated by at least one of the plurality of call devices based on the translation capability to convert the first text information into at least one language of the language category set.

[0057] A call device within the call device group and configured to translate the first text information is referred to as the translative call device. Here, when there are multiple language categories in the language category set, there may be multiple language categories of the second text information correspondingly. For example, the language category of the second text information may cover all language categories in the language category set or include all language categories in the language category set other than the language category of the first text information.

[0058] For example, when the first text information is Chinese and the language category set includes Chinese, English, Japanese and French, the language categories of the second text information may be Chinese, English, Japanese and French, or English, Japanese and French (the second text information in Chinese may adopt the first text information).

[0059] The first text information is sent to a device supporting conversion of Chinese into English, whereby the device translates the first text information into second text information in English. The first text information is sent to a device supporting conversion of Chinese into Japanese, whereby the device translates the first text information into second text information in Japanese. The first text information is sent to a device supporting conversion of Chinese into French, whereby the device translates the first text information into second text information in French. The administrative call device aggregates the second text information in Japanese, the second text information in French and the second text information in Chinese so as to obtain the second text information in three language formats (the second text information in Chinese may be equivalent to the first text information in Chinese).

[0060] In one embodiment, the number of the first text information is multiple. The method includes: adding, to each piece of the first text information, a timestamp characterizing an acquisition time of the first text information. Step 104 includes: distributing the multiple pieces of first text information to a call device belonging to the call device group and supporting translation of the first text information into the language categories in the language category set based on a chronological order of the multiple timestamps in the multiple pieces of first text information and the translation capability of each call device.

[0061] Step 105: Send the second text information to at least one target call device among the plurality of call devices, wherein the language category adopted by the at least one target call device is the same as the language category of the second text information

[0062] Step 106: Obtain a second audio file, wherein the second audio file is generated by the target call device converting the second text information into an audio file of the language category adopted by the target call device.

[0063] Here, the target call device is a call device belonging to the call device group and configured to convert the second text information into the second audio file. The second audio file is typically played in speech at the target call device.

[0064] Fig. 3 is an exemplary schematic diagram of a text sorting process according to an embodiment of the present disclosure. In an embodiment, the device performing translation may add tag user identity, timestamp the text that needs to be translated and tag target language. When the mastermind distributes different translation tasks, breakpoints can be marked according to the timestamp. After translations are done, they can be merged. At the same time, the speech of different users can be arranged in an orderly manner and finally displayed into a logical language. In Fig. 3, a Chinese audio from a user at device D 340 may be acquired by device D 340 in Chinese. Device D 340 may convert, based on an ASR technology, the Chinese audio from the user at device D 340 into a Chinese text: " ", append a timestamp to the Chinese text, and then send the Chinese text with the timestamp: " " to the administrative call device. A Chinese audio from a user at device A 310 may be acquired by device A 310 in Chinese. Device A 310 may convert, based on the ASR technology, the Chinese audio from the user at device A 310 into a Chinese text: " ", append a timestamp to the Chinese text, and then send the Chinese text with the timestamp: " " to the administrative call device. An English audio from a user at device B 320 may be acquired by device B 320 in English. Device B 320 may convert, based on the ASR technology, the English audio from the user at device B 320 into an English text: "Nice to meet you", append a timestamp to the English text, and then send the English text with the timestamp: "Nice to meet you" to the administrative call device. An English audio from a user at device C 330 may be acquired by device C 330 in English. Device C 330 may convert, based on the ASR technology, the English audio from the user at device C 330 into an English text: "how long will today's meeting last", append a timestamp to the English text, and then send the English text with a timestamp: "how long will today's meeting last" to the administrative call device (assumed as device D 340 initiating the session).

[0065] In the administrative call device, the texts sent by the devices may be sorted based on the timestamp of each text. Each text may be then allocated to a respective translative call device for translation by the translative call device based on a text sorting result and the translation capability of each device.

[0066] For example, the administrative call device may send " " and " " to device B 320 having a Chinese-English translation capability, and device B 320 may append an original timestamp of " " to an English translation result "It's time for the meeting" of " ", append an original timestamp of " ？" to an English translation result "Is everyone ready for the meeting?" of " ", and return the two English translation results (containing the respective original timestamps) to the administrative call device. Furthermore, the administrative call device may send "Nice to meet you" and "how long will today's meeting last?" to device C 330 having an English-Chinese translation capability, and device C 330 may append an original timestamp of "Nice to meet you" to a Chinese translation result " " of "Nice to meet you", append an original timestamp of "how long will today's meeting last?" to a Chinese translation result " ？" of "how long will today's meeting last?", and return the two Chinese translation results (containing the respective original timestamps) to the administrative call device.

[0067] There may be one or more translative call devices allocated with translation tasks of the same language category (e.g., Chinese-English translations) in consideration of the processing capability occupation of the call devices. Furthermore, in consideration of the translation capability difference of the translative call devices, one translative call device may perform one language category translation task (e.g., Chinese-English translation) and may simultaneously perform multiple language category translation tasks (e.g., Chinese-English translation and French-English translation).

[0068] At the administrative call device, a text sequence for each language category in the language category set may be generated based on translation results of the respective translative call devices. In Fig. 3, Chinese and English text sequences may be generated (sorted by timestamps). The English text sequence contains: 1, It's time for the meeting; 2, Is everyone ready for the meeting?; 3, Nice to meet you; and 4, how long will today's meeting last?, which are sorted in timestamp sequence. The Chinese text sequence may contain: 1, ; 2, ; 3, ; and 4, , which are sorted in timestamp sequence.

[0069] The administrative call device may send the text sequence for each language category to a device (i.e., target call device) that adopts the same language category. For example, in Fig. 3, the administrative call device may send the Chinese text sequence to device A 310 and device D 340 in Chinese, and send the English text sequence to device B 320 and device C 330 in English. Device A 310 and device D 340, based on a TTS technology, respectively may convert the Chinese text sequence into a Chinese audio stream and play the Chinese audio stream. Device B 320 and device C 340, based on the TTS technology, respectively convert the English text sequence into an English audio stream and play the English audio stream.

[0070] In one embodiment, the number of the first text information may be multiple. The method includes: adding, to the each piece of the first text information, a tag of a user identity characterizing the first text information. Step 105 includes: sending the second text information to a target call device adopting the same language category as the second text information, where the corresponding call device for acquiring the first audio file is removed from the target call device based on the tag of the user identity.

[0071] For example, in Fig. 3, the text " " sent by the user at device D 340 further may carry a user tag of device D 340. Then, when the second text information is sent to device D 340, the content " " may be removed. Therefore, there is no need for a TTS conversion of " " spoken by the user at device D 340. Therefore, by removing the corresponding call device for acquiring the first audio file, repeated translation of the words spoken by the user at the device can be avoided.

[0072] In one embodiment, the method includes: determining a caller during the audio and video call as a default administrative call device.

[0073] Therefore, by determining the caller during the audio and video call as a default administrative call device, the administrative call device can be quickly set.

[0074] In one embodiment, the method includes: determining that each call device is an ordinary device or a candidate administrative device based on a comparison process between a basic capability of each call device and a predetermined threshold; assessing a professional capability of the candidate administrative device; and updating the default administrative call device based on an assessment result. Here, the basic capability may be a capability of general hardware of the call device, and the professional capability may be a capability of professional hardware of the call device. For example, the basic capability includes at least one of the following: a remaining power, a network bandwidth, a memory capacity, and the like. The professional capability includes at least one of the following: a graphic processing unit capability, a neural processing unit capability, a central processing unit capability, and the like. Therefore, by screening the call devices through the basic capability and the professional capability, devices having more capabilities can be determined as candidate administrative devices. For example, the basic capability and the professional capability of each call device may be carried in the description information. The administrative call device may determine that the call device is an ordinary device or a candidate administrative device based on the description information of each call device.

[0075] For example, capability categories may include: (1) basic capability: power, network bandwidth, and memory (ROM&RAM); and (2) professional capability, where the professional capability A of a neural processing unit has the following calculation formula: A = MAC matrix row * MAC matrix column * 2 * main frequency, the professional capability B of a central processing unit has the following calculation formula: B = CPU kernel number * single-frequency main frequency * single-cycle floating-point computing capability, and the professional capability C of a graphics processing unit has the following calculation formula: C = GPU kernel number * (single-cycle floating-point computing capability * main frequency).

[0076] While typical examples of the capability categories and the calculation formulas have been exemplarily described above, it will be appreciated by those skilled in the art that such descriptions are merely exemplary and are not intended to limit the protection scope of the embodiments of the present disclosure.

[0077] In one embodiment, assessing the professional capability of the candidate administrative device includes at least one of the following: determining a first professional capability index of a graphics processor based on a main frequency of a neural processing unit, a row of a multiply-accumulate array, and a column of the multiply-accumulate array; determining a second professional capability index of a central processing unit based on a kernel number, single-frequency main frequency and single-cycle floating-point computing capability of the central processing unit; and determining a third professional capability index of the graphics processor based on a kernel number, main frequency and single-cycle floating-point computing capability of the graphics processor.

[0078] It can be seen that based on calculation of the professional capabilities of the candidate administrative devices according to various types of indexes, the professional capabilities of the candidate administrative devices may be quantified, thereby laying the foundation for the subsequent assessment process.

[0079] In one embodiment, assessing a professional capability of the candidate administrative device includes: determining a significance matrix characterizing a significance between any two of the first professional capability index, the second professional capability index and the third professional capability index; determining a largest eigenvalue of the significance matrix, and performing a consistency check on the significance matrix based on the largest eigenvalue; calculating respective weights of the first professional capability index, the second professional capability index and the third professional capability index based on the significance matrix passing the consistency check; and determining an assessment result of the professional capability of the candidate administrative device based on the respective weights of the first professional capability index, the second professional capability index and the third professional capability index, and a predetermined reference model.

[0080] Therefore, the candidate administrative devices may be updated in a timely manner through the professional capability assessment results.

[0081] In one embodiment, distributing, based on the translation capability of each call device, the first text information to a translative call device belonging to the call device group and supporting translation of the first text information into the language categories in the language category set includes: distributing, in a case that there are multiple translative call devices that support translation of the first text information into the same language category and a corresponding call device for acquiring the first audio file supports the translation of the first text information into the same language category, the first text information to the corresponding call device for acquiring the first audio file. Therefore, the first text information is distributed to the corresponding call device for acquiring the first audio file for translation, thereby improving translation efficiency.

[0082] Fig. 4 is a block diagram of an administrative call device according to an embodiment of the present disclosure. Fig. 5 is an exemplary schematic diagram of a cross-language call process according to an embodiment of the present disclosure.

[0083] As shown in Fig. 4 and Fig. 5, the administrative call device 400 may include: a data processing module 410, a decision module 420, a monitoring module 430, a storage module 440, and a communication module 450. A device initiating an audio and video session may be determined as the administrative call device 400 by default. The capabilities of devices added later may be collected. The administrative call device 400 may classify the devices, according to basic capabilities, into an ordinary user group and a candidate administrative call device group, and then score and sort (in descending order) the devices according to the capabilities of the candidate administrative call device group. When an audio and video call is established, the administrative call device 400 may collect texts to be translated provided by a user, and distributes translation tasks according to the capabilities of the devices in the candidate administrative call device group. The devices distributed with the translation tasks may perform translation based on respective large language models (LLM). After translation, the administrative call device 400 may integrate the translated texts according to timestamps and returns the texts to corresponding devices. Finally, the corresponding devices may convert the translated texts into audio according to a TTS technology.

[0084] The monitoring module 430 in the administrative call device 400 is responsible for monitoring the changes in the basic capabilities of the devices and the performance change of the whole system. If necessary, the administrative call device 400 is replaced or the candidate administrative call device group is updated. The implementation of each module will be described in detail below.

[0085] (1) Data Processing Module 410

[0086] The main functions of the data processing module 410 include: voiceprint recognition, data collection and data marking.

[0087] Voiceprint Recognition: In the embodiments of the present disclosure, a Gaussian mixture model-universal background model (GMM-UBM) may be used for multi-user marking, and speech contents correspond to a user identity, so as to achieve the purpose of extracting a speech of a target person from an input speech.

[0088] Data Collection: The data collection is divided into capability collection and demand collection, where the capability collection is to collect data according to the basic capabilities and professional capabilities of user device, and the demand collection is to collect data according to the translation demands of the user and a language format adopted thereby. Subsequently, the administrative call device distributes the translation tasks according to the capabilities of different devices in the candidate administrative call device group.

[0089] (1) Demand Collection: Each device accessing an audio and video call first converts an audio file into text through an automatic speech recognition (ASR) server, and then transmits the text to the administrative call device. The administrative call device collects the amount of bytes to be processed within a certain time by each device accessing the audio and video call, whereby translation tasks may be distributed according to demands and the capabilities of each device.

[0090] (2) Capability Classification: Capability categories may include: (2.1) basic capability: power, network bandwidth, and memory (ROM & RAM); and (2.2) professional capability, where the professional capability A of a neural processing unit has the following calculation formula: A = MAC matrix row * MAC matrix column * 2 * main frequency, the professional capability B of a central processing unit has the following calculation formula: B = CPU kernel number * single-frequency main frequency * single-cycle floating-point computing capability, and the professional capability C of a graphics processing unit has the following calculation formula: C = GPU kernel number * (single-cycle floating-point computing capability * main frequency).

[0091] (3) Definition of Signaling in the process of During Audio and Video call: Here, an SDP data format is extended, and a capability description field is added to indicate a translation capability of a device and a language format adopted thereby.

[0092] (4) Data Marking: An identity (ID) of the device is marked before the user translates based on the LLM. The text information of audio translation is appended the timestamp, and the translated target language is also marked. Therefore, when the administrative call device distributes different translation tasks, breakpoint marking may be performed according to a timestamp, and the translations may be combined after the translation tasks are completed. At the same time, speeches from different devices may be arranged in an orderly manner and finally displayed as a logical speech.

[0093] (2) Monitoring Module 430

[0094] The monitoring module 430 may be configured to:

[0095] (1) Monitor changes in the basic capabilities of the candidate administrative call device group (e.g. low power or offline): if the administrative call device changes, the administrative call device is replaced based on the candidate administrative call device group, and the candidate administrative call device group is dynamically updated in other situations.

[0096] (2) Monitor overall performance change: if the translation capability is insufficient, the administrative call device is replaced based on the candidate administrative call device group.

[0097] (3) Decision Module 420

[0098] The decision module 420 may be configured to:

[0099] (1) Basic conditions screening: grouping is performed according to the basic capability (power, network and memory) of each device, and when the basic capability of a device is lower than a reference condition, it is determined that the device belongs to the ordinary device group, otherwise, the device belongs to the candidate administrative call device group.

[0100] (2) Scoring mechanism: the professional capability of each user is scored based on an analytic hierarchy process. The scoring mechanism specifically includes: (2.1) The professional capability is analyzed through a control variable method to determine a significance matrix characterizing a significance between any two of a first professional capability index, a second professional capability index and a third professional capability index. For example, a 3-row and 3-column significance matrix A=(aij) is generated, where aijis the significance of an ithprofessional capability index relative to a jthprofessional capability index, and the values of i and j are within [1, 3]. (2.2) A largest eigenvalue of the significance matrix A is determined, and a consistency check is performed on the significance matrix based on the largest eigenvalue to obtain a consistency ratio (CR), where when CR is less than 0.1, the consistency check is considered to be passed. (2.3) Respective weights of the first professional capability index, the second professional capability index and the third professional capability index are calculated based on the significance matrix passing the consistency check. (2.4) An assessment result of the professional capability of the candidate administrative device is determined based on the respective weights of the first professional capability index, the second professional capability index and the third professional capability index, and a predetermined reference model.

[0101] While the scoring mechanism is described above by taking the analytic hierarchy process as an example, it will be appreciated by those skilled in the art that such descriptions are merely exemplary and are not intended to limit the protection scope of the embodiments of the present disclosure.

[0102] (3) Data Distribution: The administrative call device distributes translation tasks according to the scoring proportion of each device in the candidate administrative call device group. The following distribution principles may be included: (1) A translation task is preferably processed by a corresponding device. (2) Tasks are preferably distributed according to an adept language of a processing end.

[0103] (4) Data Combination: The translated data is combined according to breakpoints, and is returned to corresponding devices according to identities of the devices.

[0104] Fig. 6 is an exemplary structural diagram of a cross-language call system according to an embodiment of the present disclosure. The cross-language call system includes: a media server 620 configured to transmit media data in a call device group 630.

[0105] The call system further includes a signaling server 610, configured to receive, in a signaling connection during an audio and video call of the call device group 630, description information of each call device based on a session description protocol message, where the description information is carried in a description field of the session description protocol message, and the description information characterizes language categories adopted by the call devices and translation capabilities of the call devices. The call device group 630 includes multiple call devices, and the multiple call devices include an administrative call device 400. The administrative call device 400 is configured to: determine a language category set of the call device group 630 based on the description information of each call device; determine first text information, where the first text information is obtained based on converting a first audio file during the audio and video call of the call device group 630, and the first audio file is converted into the first text information at a corresponding call device for acquiring the first audio file based on the language category adopted by the corresponding call device; distribute, based on the translation capability of each call device, the first text information to a call device belonging to the call device group 630 and supporting translation of the first text information into the language categories in the language category set, whereby the call device translates the first text information into second text information, where the language category of the second text information belongs to the language category set; and send the second text information to a target call device in the call device group 630, where the language category adopted by the target call device is the same as the language category of the second text information, whereby the target call device converts the second text information into a second audio file, and the language category of the second audio file is the language category adopted by the target call device.

[0106] In one embodiment, the number of the first text information is multiple. The administrative call device 400 is configured to: add, to each piece of the first text information, a tag of a user identity characterizing the first text information; and send the second text information to a target call device adopting the same language category as the second text information, where the corresponding call device for acquiring the first audio file is removed from the target call device based on the tag of the user identity.

[0107] In summary, the embodiment of the present disclosure may provide a cross-language real-time call method. A concept of groups may be established for device sides having the translation capability, which cooperate under the unified administration of an administrative call device 400 and jointly complete translation tasks finally. The embodiment of the present disclosure may be improved on the current situation where only one-to-one translation processing can be performed and may be applied to one-to-many scenarios. The embodiment of the present disclosure may be improved on the problem that the current audio call receiver can both receive an original audio and hear a translated audio which results in disorder, and the receiving end only receives the translated audio. According to the embodiment of the present disclosure, the transmitter and the receiver may transmit text information in the scenario of network video call, thereby reducing the delay and improving the user experience. According to the embodiment of the present disclosure, key user features may be extracted in a case where there are multiple people speaking around, so as to perform speech translation only for key users, thereby avoiding a situation of disordered multi-person translation. According to the embodiment of the present disclosure, device usage may be dynamically monitored. If the basic capability of the administrative call device 400 is not satisfied, dynamic adjustment is performed to improve the user experience.

[0108] Fig. 7A is an exemplary schematic diagram of a one-to-one audio and video call according to an embodiment of the present disclosure. In Fig. 7A, for a one-to-one video call, capabilities of both devices are collected, and both devices are classified based on the capabilities. A caller 710a on the left side may belong to a candidate administrative device group, and a callee 720a on the right side may belong to an ordinary device group. A target language to be translated may be then selected. Since only the caller 710a currently supports a translation function, translation may be performed by the caller 710a, and the translated text may be returned to the callee 720a.

[0109] Fig. 7B is an exemplary schematic diagram of a one-to-many audio and video call according to an embodiment of the present disclosure. In Fig. 7B, a one-to-many video call scenario is described, in which a device may not have translation capabilities and multiple languages may be required to be translated. First, capability collection may be performed. A caller 710b may be defaulted as an administrative call device, which collects and scores the capabilities of devices when establishing a call connection, where the score of the caller 710b is 100, the score of callee1 720b is 100, the score of callee2 730b is 120, and the score of callee3 740b is 50. According to score results for device capabilities, the caller 710b, callee1 720b and callee2 730b may be classified into a candidate administrative device group, and callee3 740b may be classified into an ordinary device group. The administrative call device (caller 710b) may collect user requirements. Next, the administrative call device (caller 710b) may distribute translation tasks according to professional capabilities of callee1 720b, callee2 730b and the caller 710b. The administrative call device (caller 710b) may collect and combine all language-translated texts. The administrative call device (caller 710b) may return the translated texts to corresponding devices. Each device may convert the text into a speech by using respective TTS capabilities.

[0110] Fig. 7C is an exemplary schematic diagram of a one-to-many audio and video call and requiring replacement of an administrative call device in accordance with an embodiment of the present disclosure. In Fig. 7C, a one-to-many video call scenario is described, where multiple devices may not have translation capabilities, and the administrative call device may be required to be replaced. First, capability collection may be performed, where a caller 710c is defaulted as an administrative call device, which collects and scores the capabilities of devices when establishing a call connection. The score of the caller 710c is 50, the score of callee1 720c is 40, the score of callee2 730c is 100, and the score of callee3 740c is 80. Classification may be performed according to device capabilities, callee2 730c and callee3 740c may be classified into a candidate administrative device group, and the caller 710c and callee1 720c may be classified into an ordinary device group. It can be seen that the default administrative call device (caller 710c) does not have the capability. Therefore, the administrative call device may be required to be replaced, and callee2 730c with the highest score in the candidate administrative device group may be selected as the updated administrative call device. The updated administrative call device (callee2 730c) may collect user requirements and distributes translation tasks according to professional capabilities of callee2 730c and callee3 740c. Callee2 730c may collect translated texts, combines the texts, and returns the translated texts to corresponding devices. Each device may convert the text into a speech by using respective TTS capabilities.

[0111] Fig. 7D is an exemplary schematic diagram of dynamically monitoring a device state change according to an embodiment of the present disclosure. In Fig. 7D, state changes may be monitored dynamically. First, capability collection may be performed, where a caller 710d is defaulted as an administrative call device, which collects and scores the capabilities of devices when establishing a call connection. The score of the caller 710d is 100, the score of callee1 720d is 60, and the score of callee2 730d is 120. Classification may be performed according device capabilities, the caller 710d, callee1 720d and callee2 730d are classified into a candidate administrative device group. The monitoring module may monitor a change in the basic capability of callee1 720d from having translation capability to not having translation capability, and classify callee1 720d into an ordinary user group. The caller 710d, serving as the administrative call device, may collect user requirements and distribute translation tasks according to professional capabilities of callee2 730d and the caller 710d. The caller 710d may collect translated texts, combine the texts, and return the translated texts to corresponding devices. Each device may convert the text into a speech by using respective TTS capabilities.

[0112] Fig. 8 is an exemplary schematic diagram of a cross-language call apparatus according to an embodiment of the present disclosure. An apparatus 800 is contained in an administrative call device in a call device group. The apparatus 800 includes: a first determination module 801, configured to determine description information of each call device in the call device group, where the description information characterizes language categories adopted by the call device and translation capabilities of the call devices; a second determination module 802, configured to determine a language category set of the call device group based on the language category adopted by each call device; a third determination module 803, configured to determine first text information, where the first text information is obtained based on converting a first audio file during an audio and video call of the call device group, and the first audio file is converted into the first text information at a corresponding call device for acquiring the first audio file based on the language category adopted by the corresponding call device; a distribution module 804, configured to distribute, based on the translation capability of each call device, the first text information to a translative call device belonging to the call device group and supporting translation of the first text information into the language categories in the language category set, whereby the translative call device translates the first text information into second text information, where the language category of the second text information belongs to the language category set; and a sending module 805, configured to send the second text information to a target call device in the call device group, where the language category adopted by the target call device is the same as the language category of the second text information, whereby the target call device converts the second text information into a second audio file, the language category of the second audio file being the language category adopted by the target call device.

[0113] In one embodiment, the number of the first text information is multiple. The third determination module 803 is configured to add, to each piece of the first text information, a timestamp characterizing an acquisition time of the first text information. The distribution module 804 is configured to distribute the multiple first text information to a translative call device belonging to the call device group and supporting translation of the first text information into the language categories in the language category set based on a chronological order of the multiple timestamps in the multiple first text information and the translation capability of each call device.

[0114] In one embodiment, the number of the first text information is multiple. The third determination module 803 is configured to add, to each piece of the first text information, a tag of a user identity characterizing the first text information. The sending module 805 is configured to send the second text information to a target call device adopting the same language category as the second text information, where the corresponding call device for acquiring the first audio file is removed from the target call device based on the tag of the user identity.

[0115] In one embodiment, the first determination module 801 is configured to receive, in a signaling connection during the audio and video call, description information of each call device based on a session description protocol message, where the description information is carried in a description field of the session description protocol message.

[0116] In one embodiment, a caller during the audio and video call is a default administrative call device.

[0117] In one embodiment, the first determination module 801 is configured to: determine that each call device is an ordinary device or a candidate administrative device based on a comparison process between a basic capability of each call device and a predetermined threshold, where the basic capability includes at least one of the following: a remaining power, a network bandwidth and a memory capacity; assess a professional capability of the candidate administrative device; and update the default administrative call device based on an assessment result, where the professional capability includes at least one of the following: a graphic processor capability, a neural processing unit capability and a central processing unit capability.

[0118] In one embodiment, the first determination module 801 is configured to perform at least one of the following: determine a first professional capability index of a graphics processor based on a main frequency of a neural processing unit, a row of a multiply-accumulate array, and a column of the multiply-accumulate array; determine a second professional capability index of a central processing unit based on a kernel number, single-frequency main frequency and single-cycle floating-point computing capability of the central processing unit; and determine a third professional capability index of the graphics processor based on a kernel number, main frequency and single-cycle floating-point computing capability of the graphics processor.

[0119] In one embodiment, the first determination module 801 is configured to: determine a significance matrix characterizing a significance between any two of the first professional capability index, the second professional capability index and the third professional capability index; determine a largest eigenvalue of the significance matrix, and perform a consistency check on the significance matrix based on the largest eigenvalue; calculate respective weights of the first professional capability index, the second professional capability index and the third professional capability index based on the significance matrix passing the consistency check; and determine an assessment result of the professional capability of the candidate administrative device based on the respective weights of the first professional capability index, the second professional capability index and the third professional capability index, and a predetermined reference model.

[0120] In one embodiment, the distribution module 804 is configured to distribute, in a case that there are multiple translative call devices that support translation of the first text information into the same language category and a corresponding call device for acquiring the first audio file supports the translation of the first text information into the same language category, the first text information to the corresponding call device for acquiring the first audio file.

[0121] Embodiments of the present disclosure also provide an electronic device having a processor-memory architecture. Fig. 9 is a structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in Fig. 9, the electronic device includes a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901. The computer program, when executed by the processor 901, implements the cross-language call method as described in any one of the above. The memory 902 may be specifically implemented as various storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory, and a programmable program read-only memory (PROM). The processor 901 may be implemented as including one or more central processing units or one or more field programmable gate arrays. The field programmable gate arrays are integrated with one or more central processing unit cores. Specifically, the central processing unit or central processing unit core can be implemented as a CPU, an MCU, or a DSP.

[0122] It should be noted that not all the steps and modules in the above flowcharts and structural diagrams are necessary, and some steps or modules may be omitted according to actual requirements. The order of execution of the steps is not fixed and may be adjusted as required. The division of various modules is merely to facilitate the description of the functional division adopted. In an actual implementation, one module may be divided into multiple modules, the functions of the multiple modules may also be realized by the same module, and these modules may be located in the same device or in different devices.

[0123] Hardware modules in various embodiments may be implemented mechanically or electronically. For example, one hardware module may include a specially designed permanent circuit or logic device (for example, a dedicated processor such as an FPGA or an ASIC) for performing a particular operation. The hardware module may also include a programmable logic device or circuit (for example, including a general purpose processor or other programmable processors) temporarily configured by software for performing a particular operation. The implementation of the hardware module mechanically, or using a dedicated permanent circuit, or using a temporarily configured circuit (for example, configured by software) may be determined based on cost and time considerations.

[0124] The present disclosure also provides a machine-readable storage medium storing instructions for causing a machine to perform the method as described herein. Specifically, a system or apparatus equipped with a storage medium may be provided. A software program code that realizes the functions of any one embodiment in the above examples is stored in the storage medium, and a computer (or a CPU or an MPU) of the system or apparatus is caused to read out and execute the program code stored in the storage medium. Furthermore, some or all of actual operations may be completed by an operating system or the like operating on the computer through instructions based on the program code. The program code read out from the storage medium may also be written into a memory provided in an expansion board inserted into the computer or into a memory provided in an expansion unit connected to the computer. Then, the instructions based on the program code cause the CPU or the like installed on the expansion board or the expansion unit to perform some or all of the actual operations, so as to realize the functions of any one of the above embodiments. Embodiments of the storage medium for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (for example, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Optionally, the program code may be downloaded from a server computer or cloud via a communication network.

[0125] As used herein, "schematic" means "serving as an instance, example, or description", and any illustration and embodiment described herein as "schematic" should not be construed as a more preferred or advantageous technical solution. For sake of clarity of the drawings, only portions of the drawings related to the present disclosure are schematically shown and are not representative of an actual structure of a product. In addition, for sake of clarity of the drawings and ease of understanding, only one of members having the same structure or function may be schematically shown or marked in some of the drawings. As used herein, "one" does not mean to limit the number of related portions of the present disclosure to "only one", and "one" does not mean to exclude the case that the number of related portions of the present disclosure is "more than one". As used herein, "upper", "lower", "front", "back", "left", "right", "inner", "outer", and the like are used merely to indicate relative positional relationships between related portions, and do not limit absolute positions of these related portions.

[0126] The above description is merely preferred examples of the present disclosure and is not intended to limit the scope of protection of the present disclosure. Any modifications, equivalent substitutions, improvements, and the like made within the spirit and principles of the present disclosure are intended to be included within the scope of protection of the present disclosure.

[0127] In one embodiment, the number of the first text information is multiple.

[0128] In one embodiment, the method may include adding, to each piece of the first text information, a timestamp characterizing an acquisition time of the first text information.

[0129] In one embodiment, the method may include adding, to each piece of the first text information, a tag of a user identity characterizing the first text information.

[0130] In one embodiment, the obtaining second text information may be based on a chronological order of the timestamps.

[0131] In one embodiment, the sending the second text information to at least one target call device may be removed from the target call device based on the tag of the user identity.

[0132] In one embodiment, the determining description information of the plurality of call devices may include receiving, in a signaling connection during the call, description information of the plurality of call devices based on a session description protocol message, where the description information is carried in a description field of the session description protocol message.

[0133] In one embodiment, the method may include determining a caller during the call as a default administrative call device.

[0134] In one embodiment, the method may include determining that at least one call device among the plurality of call devices is an ordinary device or a candidate administrative device based on a comparison process between a basic capability of the plurality of call device and a predetermined threshold.

[0135] In one embodiment, the method may include assessing a professional capability of the candidate administrative device.

[0136] In one embodiment, the method may include updating the default administrative call device based on an assessment result.

[0137] In one embodiment, the assessing the professional capability of the candidate administrative device may include determining a first professional capability index of a graphics processor based on a main frequency of a neural processing unit, a row of a multiply-accumulate array, and a column of the multiply-accumulate array;

[0138] In one embodiment, the assessing the professional capability of the candidate administrative device may include determining a second professional capability index of a central processing unit based on a kernel number, single-frequency main frequency and single-cycle floating-point computing capability of the central processing unit; and

[0139] In one embodiment, the assessing the professional capability of the candidate administrative device may include determining a third professional capability index of the graphics processor based on a kernel number, main frequency and single-cycle floating-point computing capability of the graphics processing unit.

[0140] In one embodiment, the assessing the professional capability of the candidate administrative device may include determining a significance matrix characterizing a significance between any two of the first professional capability index, the second professional capability index and the third professional capability index;

[0141] In one embodiment, the assessing the professional capability of the candidate administrative device may include determining a largest eigenvalue of the significance matrix, and performing a consistency check on the significance matrix based on the largest eigenvalue;

[0142] In one embodiment, the assessing the professional capability of the candidate administrative device may include calculating respective weights of the first professional capability index, the second professional capability index and the third professional capability index based on the significance matrix passing the consistency check; and

[0143] In one embodiment, the assessing the professional capability of the candidate administrative device may include determining an assessment result of the professional capability of the candidate administrative device based on the respective weights of the first professional capability index, the second professional capability index and the third professional capability index, and a predetermined reference model.

[0144] In one embodiment, the method may include distributing, in a case that there are multiple call devices that support translation of the first text information into the same language category and a corresponding call device for acquiring the first audio file supports the translation of the first text information into the same language category, the first text information to the corresponding call device for acquiring the first audio file.

[0145] In one embodiment, at least one processor may cause the electronic device to add, to each piece of the first text information, a timestamp characterizing an acquisition time of the first text information.

[0146] In one embodiment, at least one processor may cause the electronic device to add, to each piece of the first text information, a tag of a user identity characterizing the first text information.

[0147] In one embodiment, to obtain second text information, the at least one processor may cause the electronic device to be based on a chronological order of the timestamps.

[0148] In one embodiment, to send the second text information to at least one target call device, the at least one processor cause the electronic device to be removed from the target call device based on the tag of the user identity.

[0149] In one embodiment, to determine description information of the plurality of call devices, the at least one processor may cause the electronic device to receive, in a signaling connection during the call, description information of the plurality of call devices based on a session description protocol message, wherein the description information is carried in a description field of the session description protocol message.

[0150] In one embodiment, the at least one processor may cause the electronic device to determine a caller during the call as a default administrative call device.

[0151] In one embodiment, the at least one processor may cause the electronic device to determine that at least one call device among the plurality of call devices is an ordinary device or a candidate administrative device based on a comparison process between a basic capability of the plurality of call devices and a predetermined threshold.

[0152] In one embodiment, the at least one processor may cause the electronic device to assess a professional capability of the candidate administrative device.

[0153] In one embodiment, the at least one processor may cause the electronic device to update the default administrative call device based on an assessment result.

[0154] A cross-language call apparatus is contained in an administrative call device in a call device group. The apparatus includes:

[0155] a first determination module, configured to determine description information of each call device in the call device group, where the description information characterizes language categories adopted by the call device and translation capabilities of the call device;

[0156] a second determination module, configured to determine a language category set of the call device group based on the language category adopted by each call device;

[0157] a third determination module, configured to determine first text information, where the first text information is obtained based on converting a first audio file during an audio and video call of the call device group, and the first audio file is converted into the first text information at a corresponding call device for acquiring the first audio file based on the language category adopted by the corresponding call device;

[0158] a distribution module, configured to distribute, based on the translation capability of each call device, the first text information to a translative call device belonging to the call device group and supporting translation of the first text information into the language categories in the language category set, whereby the translative call device translates the first text information into second text information, where the language category of the second text information belongs to the language category set; and

[0159] a sending module, configured to send the second text information to a target call device in the call device group, where the language category adopted by the target call device is the same as the language category of the second text information, whereby the target call device converts the second text information into a second audio file, the language category of the second audio file being the language category adopted by the target call device.

[0160] In one embodiment, the number of the first text information is multiple.

[0161] The third determination module is configured to add, to each piece of the first text information, a timestamp characterizing an acquisition time of the first text information.

[0162] The distribution module is configured to distribute the multiple first text information to a translative call device belonging to the call device group and supporting translation of the first text information into the language categories in the language category set based on a chronological order of the multiple timestamps in the multiple first text information and the translation capability of each call device.

[0163] In one embodiment, the number of the first text information is multiple.

[0164] The third determination module is configured to add, to each piece of the first text information, a tag of a user identity characterizing the first text information.

[0165] The sending module is configured to send the second text information to a target call device adopting the same language category as the second text information, where the corresponding call device for acquiring the first audio file is removed from the target call device based on the tag of the user identity.

[0166] In one embodiment, the first determination module is configured to receive, in a signaling connection during the audio and video call, description information of each call device based on a session description protocol message, where the description information is carried in a description field of the session description protocol message.

[0167] A cross-language call system includes:

[0168] a signaling server, configured to receive, in a signaling connection during an audio and video call of a call device group, description information of each call device in the call device group based on a session description protocol message, where the description information is carried in a description field of the session description protocol message, the description information characterizes language categories adopted by the call device and translation capabilities of the call devices, the call device group includes multiple call devices, and the multiple call devices include an administrative call device.

[0169] The administrative call device, is configured to determine a language category set of the call device group based on the description information of each call device; determine first text information, where the first text information is obtained based on converting a first audio file during the audio and video call of the call device group, and the first audio file is converted into the first text information at a corresponding call device for acquiring the first audio file based on the language category adopted by the corresponding call device; distribute, based on the translation capability of each call device, the first text information to a translative call device belonging to the call device group and supporting translation of the first text information into the language categories in the language category set, whereby the translative call device translates the first text information into second text information, where the language category of the second text information belongs to the language category set; and send the second text information to a target call device in the call device group, where the language category adopted by the target call device is the same as the language category of the second text information, whereby the target call device converts the second text information into a second audio file, and the language category of the second audio file is the language category adopted by the target call device.

[0170] In one embodiment, the number of the first text information is multiple.

[0171] The administrative call device is configured to: add, to each piece of the first text information, a tag of a user identity characterizing the first text information; and send the second text information to a target call device adopting the same language category as the second text information, where the corresponding call device for acquiring the first audio file is removed from the target call device based on the tag of the user identity.

[0172] A program product includes a computer program that, when executed by a processor, implements the cross-language call method as described in any one of the above.

Claims

1.A method performed by a plurality of call devices, comprising:determining description information of the plurality of call devices, wherein the description information comprises language category adopted by the plurality of call devices and translation capability of the plurality of call devices;determining a language category set of the plurality of the call devices based on the language category adopted by the plurality of call devices;obtaining first text information, wherein the first text information is converted based on a first audio file obtained from a call among the plurality of call devices;obtaining second text information, wherein the second text information is translated by at least one of the plurality of call devices based on the translation capability to convert the first text information into at least one language of the language category set;sending the second text information to at least one target call device among the plurality of call devices, wherein the language category adopted by the at least one target call device is the same as the language category of the second text information; andobtaining a second audio file, wherein the second audio file is generated by the target call device converting the second text information into an audio file of the language category adopted by the target call device.2.The method according to claim 1, further comprising:adding, to each piece of the first text information, a timestamp characterizing an acquisition time of the first text information; andadding, to each piece of the first text information, a tag of a user identity characterizing the first text information;3.The method according to claim 2, wherein the obtaining second text information is based on a chronological order of the timestamps,wherein the sending the second text information to at least one target call device is removed from the target call device based on the tag of the user identity.4.The method according to any one of claims 1 to 3, wherein the determining description information of the plurality of call devices comprises:receiving, in a signaling connection during the call, description information of the plurality of call devices based on a session description protocol message, wherein the description information is carried in a description field of the session description protocol message.5.The method according to any one of claims 1 to 4, further comprising:determining a caller during the call as a default administrative call device.6.The method according to claim 5, further comprising:determining that at least one call device among the plurality of call devices is an ordinary device or a candidate administrative device based on a comparison process between a basic capability of the plurality of call devices and a predetermined threshold;assessing a professional capability of the candidate administrative device; andupdating the default administrative call device based on an assessment result.7.The method according to claim 6, wherein the assessing the professional capability of the candidate administrative device comprises at least one of the following:determining a first professional capability index of a graphics processor based on a main frequency of a neural processing unit, a row of a multiply-accumulate array, and a column of the multiply-accumulate array;determining a second professional capability index of a central processing unit based on a kernel number, single-frequency main frequency and single-cycle floating-point computing capability of the central processing unit; anddetermining a third professional capability index of the graphics processor based on a kernel number, main frequency and single-cycle floating-point computing capability of the graphics processing unit.8.The method according to any one of claims 6 to 7, wherein the assessing the professional capability of the candidate administrative device comprises:determining a significance matrix characterizing a significance between any two of the first professional capability index, the second professional capability index and the third professional capability index;determining a largest eigenvalue of the significance matrix, and performing a consistency check on the significance matrix based on the largest eigenvalue;calculating respective weights of the first professional capability index, the second professional capability index and the third professional capability index based on the significance matrix passing the consistency check; anddetermining an assessment result of the professional capability of the candidate administrative device based on the respective weights of the first professional capability index, the second professional capability index and the third professional capability index, and a predetermined reference model.9.An electronic device, comprisingmemory storing instructions;at least one processor including processing circuitry, memory storing instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to:determine description information of the plurality of call devices, wherein the description information comprises language category adopted by the plurality of call devices and translation capability of the plurality of call devices;determine a language category set of the plurality of call devices based on the language category adopted by the plurality of call devices;obtain first text information, wherein the first text information is converted based on a first audio file obtained from a call among the plurality of call devices;obtain second text information, wherein the second text information is translated by at least one of based on the translation capability to convert the first text information into at least one language category; andsend the second text information to at least one target call device among the plurality of call devices, wherein the language category adopted by the at least one target call device is the same as the language category of the second text information.10.The electronic device according to claim 9, at least one processor cause further the electronic device to:add, to each piece of the first text information, a timestamp characterizing an acquisition time of the first text information; andadd, to each piece of the first text information, a tag of a user identity characterizing the first text information.11.The electronic device according to claim 10,wherein, to obtain second text information, the at least one processor cause the electronic device to be based on a chronological order of the timestamps; andwherein, to send the second text information to at least one target call device, the at least one processor cause the electronic device to be removed from the target call device based on the tag of the user identity.12.The electronic device according to any one of claims 9 to 11,wherein, to determine description information of the plurality of call devices, the at least one processor cause the electronic device to receive, in a signaling connection during the call, description information of the plurality of call devices based on a session description protocol message, wherein the description information is carried in a description field of the session description protocol message.13.The electronic device according to any one of claims 9 to 12, the at least one processor cause the electronic device to determine a caller during the call as a default administrative call device.14.The electronic device according to claim 13, the at least one processor further cause the electronic device to:determine that at least one call device among the plurality of call devices is an ordinary device or a candidate administrative device based on a comparison process between a basic capability of the plurality of call devices and a predetermined threshold;assess a professional capability of the candidate administrative device; andupdate the default administrative call device based on an assessment result.15.A computer-readable storage medium containing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform the cross-language call method according to any one of claims 1 to 8.

Images