Translation service processing method during video call

Abstract

The invention discloses a translation service processing method during a video call, and belongs to the technical field of communication. The method is applied to an application server (AS), and comprises the following steps: determining to establish a call link between a first terminal and a second terminal, and obtaining a number receiving result of a target terminal; acquiring an audio stream receiving address, a subtitle receiving address and a media capability platform control address under the condition of determining that the target terminal starts the target translation service based on the number receiving result; and translation service indication information is sent to the media capability platform indicated by the media capability platform control address, and the translation service indication information is used for indicating the media capability platform to acquire the audio copy stream based on the audio stream receiving address and execute processing corresponding to the target translation service on the audio copy stream. And sending the subtitle data obtained after processing to a target terminal corresponding to the subtitle receiving address. Through the mode, efficient and reliable media stream translation service can be realized, and the processing efficiency of the audio stream translation service is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technologies, and in particular to a method for processing translation services during a video call. Background Art

[0002] Video calling is a communication method based on the internet and mobile internet, which transmits human voice and images in real time between terminals such as mobile phones and computers. Compared with traditional calls on 2G, 3G, and 4G networks, 5G calls offer faster data transmission speeds and clearer image quality. 5G calls can also add some fun features to video calls, such as intelligent translation, fun calls, intelligent customer service, content sharing, and remote assistance. However, advanced features such as intelligent translation still need to be improved.

[0003] Currently, multimedia video call services typically involve complex network element processing technologies, especially when the network architecture is distributed across multiple geographic regions or provinces. Each region or province is equipped with an independent intelligent translation application server (AS), making it difficult to achieve unified business logic and standardized processes. Moreover, if new services or functions are to be added, the intelligent translation AS in each region or province needs to be deployed and configured accordingly, increasing operational complexity and the risk of errors. Summary of the Invention

[0004] The embodiment of the present application provides a method for processing translation services during video calls, so as to at least solve the problem that related technologies are difficult to achieve unified business logic and standardized processes, and have high operation and maintenance difficulties and high error risks.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In the first aspect, an embodiment of the present application provides a method for processing translation services during a video call, which is applied to an application server AS, including: determining that a call link is established between a first terminal and a second terminal, and obtaining a number collection result of a target terminal; wherein the target terminal includes the first terminal and the second terminal; when it is determined based on the number collection result that the target terminal starts a target translation service, obtaining an audio stream receiving address, a subtitle receiving address, and a media capability platform control address; sending translation service indication information to the media capability platform indicated by the media capability platform control address, wherein the translation service indication information is used to instruct the media capability platform to obtain an audio copy stream based on the audio stream receiving address, and perform processing corresponding to the target translation service on the audio copy stream, and send the subtitle data obtained after processing to the target terminal corresponding to the subtitle receiving address.

[0007] In the second aspect, an embodiment of the present application provides a method for processing translation services during a video call, which is applied to a target terminal, including: initiating a video call and establishing a data channel with a unified media processing network element UMF; obtaining and downloading a translation application from the unified media processing network element UMF through the data channel; the translation application is embedded with translation service subscription information corresponding to the target terminal; obtaining an interactive operation based on the translation application, and sending the interactive operation to the UMF when it is determined that the translation service corresponding to the interactive operation is available according to the translation service subscription information; receiving subtitle data sent by the UMF, synthesizing the subtitle data with the media stream corresponding to the interactive operation, and displaying the synthesized video data; wherein, the subtitle data is sent to the UMF by the media capability platform based on the received translation service indication information.

[0008] In a third aspect, an embodiment of the present application provides an application server, comprising: a number collection and acquisition module, used to determine whether a call link is established between a first terminal and a second terminal, and obtain the number collection result of a target terminal; wherein the target terminal includes the first terminal and the second terminal; an address acquisition module, used to obtain an audio stream receiving address, a subtitle receiving address and a media capability platform control address when it is determined based on the number collection result that the target terminal turns on a target translation service; an information sending module, used to send translation service indication information to the media capability platform indicated by the media capability platform control address, wherein the translation service indication information is used to instruct the media capability platform to obtain an audio copy stream based on the audio stream receiving address, and perform processing corresponding to the target translation service on the audio copy stream, and send the subtitle data obtained after processing to the target terminal corresponding to the subtitle receiving address.

[0009] In a fourth aspect, an embodiment of the present application provides a terminal, including: a channel establishment module, used to initiate a video call and establish a data channel with a unified media processing network element UMF; a program download module, used to obtain and download a translation application from the unified media processing network element UMF through the data channel; the translation application is embedded with translation service subscription information corresponding to the target terminal; an operation sending module, used to obtain an interactive operation based on the translation application, and when it is determined that the translation service corresponding to the interactive operation is available according to the translation service subscription information, the interactive operation is sent to the UMF; a data receiving module, used to receive subtitle data sent by the UMF, synthesize the subtitle data with the media stream corresponding to the interactive operation, and display the synthesized video data; wherein, the subtitle data is sent to the UMF by the media capability platform based on the received translation service indication information.

[0010] In a fifth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first or second aspect above are implemented.

[0011] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first or second aspect above are implemented.

[0012] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the method described in the first or second aspect above.

[0013] In an embodiment of the present application, the application server AS determines that a call link is established between the first terminal and the second terminal, and obtains the number collection result of the target terminal; when it is determined based on the number collection result that the target terminal has started the target translation service, the application server AS obtains the audio stream receiving address, the subtitle receiving address, and the media capability platform control address; and sends translation service indication information to the media capability platform indicated by the media capability platform control address. The translation service indication information is used to instruct the media capability platform to obtain the audio copy stream based on the audio stream receiving address, and perform processing corresponding to the target translation service on the audio copy stream, and send the subtitle data obtained after processing to the target terminal corresponding to the subtitle receiving address. In this way, through standardized interfaces and unified business logic, an efficient and reliable media stream translation service can be implemented, the processing efficiency of the audio stream translation service can be improved, and it is more scalable and maintainable.

[0014] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] Figure 1 It shows a schematic diagram of the structure of a general network in the prior art;

[0017] Figure 2 A schematic diagram showing the structure of the network provided in the embodiment of the present application is shown;

[0018] Figure 3A flow chart showing a method for processing translation services during a video call provided by an embodiment of the present application is shown;

[0019] Figure 4 A schematic diagram showing a flow chart of a method for presetting service data on an application server AS provided in an embodiment of the present application is shown;

[0020] Figure 5 Another flowchart of a method for processing translation services during a video call provided by an embodiment of the present application is shown;

[0021] Figure 6 Another flowchart of a method for processing translation services during a video call provided by an embodiment of the present application is shown;

[0022] Figure 7 Another flowchart of a method for processing translation services during a video call provided by an embodiment of the present application is shown;

[0023] Figure 8 Another flowchart of a method for processing translation services during a video call provided by an embodiment of the present application is shown;

[0024] Figure 9 Another flowchart of a method for processing translation services during a video call provided by an embodiment of the present application is shown;

[0025] Figure 10 Another flowchart of a method for processing translation services during a video call provided by an embodiment of the present application is shown;

[0026] Figure 11 A schematic diagram of the structure of the application server provided in an embodiment of the present application is shown;

[0027] Figure 12 A schematic diagram of the structure of a terminal provided in an embodiment of the present application is shown;

[0028] Figure 13 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0030] Multimedia video call services usually involve complex network element processing technology, especially when the network architecture is distributed across multiple geographical regions or provinces, each of which is equipped with an independent intelligent translation application server (AS), such as Figure 1 As shown in , each geographical area corresponds to its own AS. This approach has at least the following problems:

[0031] 1. Difficulty in standardization: Businesses may evolve continuously based on market changes and operational requirements. If a unified business logic needs to be distributed across geographical regions or provinces, expanding, developing, or adjusting the same business will incur significant communication costs, and the results may vary.

[0032] 2. Difficulty in scalability: When adding new services, additional ASs need to be deployed on the New Calling Platform (NCP) in each geographic region or province. This operation is cumbersome and greatly increases the possibility of errors.

[0033] 3. Increased operational difficulty: The same operation management platform needs to connect to multiple ASs at the same time, which increases the difficulty of issuing and maintaining user data.

[0034] Figure 2 A structural diagram of the network provided by an embodiment of the present application is shown. As shown in the figure, the network provided by an embodiment of the present application includes multiple network elements, for example, a first network element 211, a second network element 212 and a third network element 213, an application server AS220, an operation management platform 230 and a media capability platform 240; wherein each network element includes: a voice application server VoLTEAS based on long-term evolution, a unified media processing network element UMF and a new call service control platform NCP.

[0035] Compared to Figure 1 The general networking shown in this application embodiment will Figure 1 The AS network elements belonging to each geographical area are extracted into a new unified application server AS. The application server AS interacts with the NCP of each geographical area to simplify the overall translation business process and network structure.

[0036] Figure 3 A flow chart of a method for processing translation services during a video call provided by an embodiment of the present application is shown. The method for processing translation services during a video call can be applied to Figure 2 As shown in the figure, the method 300 may include the following steps:

[0037] Step 301: Determine whether a call link is established between a first terminal and a second terminal, and obtain a number collection result of a target terminal.

[0038] The target terminal includes the first terminal and the second terminal, and the number collection result is determined by the NCP based on a user operation instruction of the target terminal.

[0039] In a specific implementation, the first terminal can be the calling terminal, and the second terminal can be the called terminal; the first terminal can also be the called terminal, and the second terminal can be the calling terminal. The calling terminal, UE A, initiates a video call request, and the called terminal, UE B, responds to the video call. A call link is established between the calling and called terminals. At this point, the application server, AS, obtains the target terminal's number collection result and, based on this result, determines whether the target terminal requires the translation service.

[0040] Step 302: When it is determined based on the number collection result that the target terminal starts the target translation service, an audio stream receiving address, a subtitle receiving address, and a media capability platform control address are obtained.

[0041] Among them, the target translation service can be speech transcription or subtitle translation; speech transcription is to convert the speech content in the video or audio into subtitle text, and subtitle translation is to translate the transcribed subtitle text into the target language; the media capability platform control address can be the address of a specified media capability platform, or it can be the address of a media capability platform determined according to the geographical location of the target terminal.

[0042] Step 303: Send translation service indication information to the media capability platform indicated by the media capability platform control address, wherein the translation service indication information is used to instruct the media capability platform to obtain the audio copy stream based on the audio stream receiving address, and perform processing corresponding to the target translation service on the audio copy stream, and send the processed subtitle data to the target terminal corresponding to the subtitle receiving address.

[0043] In a specific implementation, the application server AS sends translation service indication information to the media capability platform indicated by the media capability platform control address, and the translation service indication information may include an audio stream receiving address and a subtitle receiving address; after the media capability platform obtains the translation service indication information, it obtains the audio copy stream based on the audio stream receiving address, and performs processing corresponding to the target translation service on the audio copy stream, and sends the processed subtitle data to the target terminal corresponding to the subtitle receiving address.

[0044] In this way, the data exchange and indication information between the target terminal, AS and media capability platform all follow a unified business logic, thereby achieving efficient and reliable media stream translation services, improving the processing efficiency of audio stream translation services, and being more scalable and maintainable.

[0045] In one possible implementation, in step 301, determining to establish a call link between the first terminal and the second terminal includes:

[0046] Step 3011: Obtain a call request event of the first terminal and a call answer event of the second terminal sent by a new call service control platform (NCP), and send a service control message to the NCP; wherein the service control message is used to instruct the NCP to anchor the audio and video media streams to the unified media processing network element (UMF);

[0047] Step 3012: Determine to establish a call link between the first terminal and the second terminal according to the audio and video anchoring control result returned by the NCP based on the service control message.

[0048] In an embodiment of the present application, by anchoring the audio and video media streams to a unified media processing network element (UMF), the audio and video streams can be centrally managed and processed, thereby ensuring call quality and consistency, and ensuring smooth and accurate calls between the first terminal and the second terminal.

[0049] In a possible implementation, in step 301, before obtaining the number collection result of the target terminal, the following steps may be further performed:

[0050] Step 3013: Send a playback request to the NCP, and the NCP controls the UMF to play the voice prompt within a specified time and return the playback result after the voice prompt is played;

[0051] Step 3014: Obtain the playback result returned by the NCP based on the playback request.

[0052] In the embodiment of the present application, before obtaining the number collection result of the target terminal sent by the NCP, the UMF is controlled by the NCP to play a voice prompt within a specified time and return the playback result after the voice prompt is played. The playback result is sent to the NCP. In this way, the target terminal can obtain the prompt information, thereby improving the user experience.

[0053] In a possible implementation, before sending the playback request to the NCP, the method further includes:

[0054] Send a system-level setting request to the operation management platform, and send the system-level setting data issued by the operation management platform based on the system-level setting request to the NCP, so that the NCP controls the UMF to play the voice prompt within the specified time according to the system-level setting data.

[0055] The system-level setting data includes system-level switch settings, prompt settings, prompt tone settings, dual-tone multi-frequency (DTMF) settings, and service default settings.

[0056] In the embodiment of the present application, before sending the playback request to the NCP, a service-level call event subscription and a user-level call event subscription may be sent to the NCP to subscribe to the call request event or call answer event of the target terminal. Figure 4 As shown in , the application server AS can preset service data in the following ways:

[0057] Step 401: The application server AS initiates a call event subscription to the NCP, subscribing to the call event corresponding to the operation performed by the user on the target terminal during the call;

[0058] Step 402: The application server AS requests the operation management platform to issue system-level settings;

[0059] Step 403: The operation management platform sends system-level setting data, including system-level switch settings, prompt settings, tone settings, DTMF settings, and service default settings, and notifies the AS of the synchronization result;

[0060] Steps 404-405: The user activates the service, triggering the operation management platform interface to synchronize the user subscription data with the NCP and the application server AS;

[0061] Step 406-Step 407: After the user activates the service, he / she sets up the service through channels such as h5. The operation management platform receives the settings and synchronizes them to the application server AS so that corresponding service processing can be performed for the user during the call.

[0062] In one possible implementation, in step 302, obtaining the audio stream receiving address, the subtitle receiving address, and the media capability platform control address includes:

[0063] Send a media stream translation request to the new call service control platform NCP, and obtain the audio stream receiving address, subtitle receiving address and media capability platform control address returned by the NCP based on the media stream translation request.

[0064] The sending of the media stream translation request to the NCP and obtaining the audio stream receiving address, subtitle receiving address, and media capability platform control address returned by the NCP based on the media stream translation request include:

[0065] Step 3021: Send a media stream replication request to the NCP. The media stream replication request is used to instruct the NCP to determine the media capability platform control address and notify the UMF to perform audio stream replication to obtain the audio stream receiving address corresponding to the UMF.

[0066] Step 3022: Acquire the media capability platform control address and the audio stream receiving address returned by the NCP based on the media stream copy request;

[0067] Step 3023: Send a subtitle stream synthesis request to the NCP, where the subtitle stream synthesis request is used to instruct the NCP to apply for subtitle stream synthesis resources from the UMF and obtain the subtitle receiving address returned by the UMF.

[0068] Step 3024: Obtain the subtitle receiving address returned by the NCP based on the subtitle stream synthesis request.

[0069] In the embodiment of the present application, by sending a media stream copy request, the NCP determines the control address of the media capability platform and informs the UMF to copy the audio stream, thereby obtaining the audio stream receiving address; by sending a subtitle stream synthesis request, the NCP applies to the UMF for subtitle stream synthesis resources and obtains the subtitle receiving address; then, the NCP returns the audio stream receiving address, the subtitle receiving address and the media capability platform control address to process subsequent translation services, ensuring the synchronous display of audio and subtitles and improving user experience.

[0070] In a possible implementation, in step 303, sending the processed subtitle data to the target terminal corresponding to the subtitle receiving address includes:

[0071] The subtitle data obtained after processing is sent to the unified media processing network element UMF indicated by the subtitle receiving address, and the subtitle data is sent to the target terminal through the UMF.

[0072] In an exemplary embodiment, Figure 5 Another flowchart of a method for processing translation services during a video call provided by an embodiment of the present application is shown. As shown in the figure, the method includes the following steps:

[0073] Steps 501-502: The calling UE A initiates a video call, and the VoLTE AS receives an INVITE request message for the video call.

[0074] Steps 503-507: After receiving the video call request message INVITE, the VoLTE AS reports the call event and related parameter information to the NCP based on the NCP subscription information. At the same time, the NCP reports the call event to the relevant AS on demand based on the subscription of each service AS.

[0075] Steps 508-513: After the video call is answered, the VoLTE AS reports the call event and audio and video media types based on the NCP. The NCP reports the answer event to the relevant AS on demand based on the service AS subscription.

[0076] Steps 514-523: The application server AS instructs the NCP to perform audio and video anchoring through a service control message. Based on the service control message from the application server AS, the NCP controls the VoLTE AS to anchor the audio and video media streams to the UMF. After the UMF is anchored to the audio and video media path, the VoLTE AS reports the UMF resources to the NCP, which then directly operates the UMF resources. The NCP reports the control results of the audio and video anchoring to the application server AS.

[0077] Steps 524-531: The application server (AS) sends a play request to the NCP, instructing the NCP to play a voice prompt. The play request carries the prompt voice information. The NCP instructs the UMF to play the voice prompt. The UMF, following the instruction, plays a one-time voice prompt to the peer user UE B within the specified duration, notifying them that their voice has been transcribed / captioned. After the voice prompt is played, the UMF sends a playback result notification to the NCP, which then sends it to the application server (AS).

[0078] Steps 532-535: The AS instructs the NCP to perform DTMF number collection, carrying the Uniform Resource Locator (URL) for the AS to receive the DTMF result; the NCP instructs the UMF to perform DTMF number collection, carrying the URL for the NCP to receive the DTMF result; the UMF will continue to detect DTMF after receiving the DTMF number collection instruction;

[0079] Step 536-Step 539: When the application server AS determines that the target terminal has started the target translation service, it sends a media stream replication request to the NCP to instruct the NCP to prepare to replicate the audio stream and optionally carry the media capability platform entry address: if the service AS carries the media capability platform entry address, the NCP gives priority to the address carried by the service AS, otherwise the NCP selects the media capability platform according to the user location information and generates the media capability platform entry address; the NCP notifies the UMF to start audio stream replication, and the media capability platform returns the audio stream receiving address and the media capability platform control address in the audio stream replication response, and returns them to the NCP at the same time, and the NCP returns them to the intelligent translation AS; the UMF starts to send the audio replication stream to the media capability platform; among which, the target translation service includes speech transcription and subtitle translation service.

[0080] Steps 540-543: The application server AS sends a subtitle stream synthesis request to the NCP to notify the NCP to prepare subtitle stream synthesis resources; the NCP applies for subtitle stream synthesis resources from the UMF, and the UMF returns the subtitle receiving address; the NCP returns the subtitle receiving address to the service AS;

[0081] Step 544: The application server AS notifies the media capability platform to start the speech transcription or subtitle translation function, carrying the subtitle receiving address and translation / transcription related information; the media capability platform sends the subtitle stream to the UMF; the UMF synthesizes the subtitle stream into the downlink video stream of the target terminal and sends it to the target terminal.

[0082] In a possible implementation, in step 303, before sending the translation service indication information to the media capability platform indicated by the media capability platform control address, the step further includes:

[0083] When it is determined that the translation service currently used by the target terminal cannot match the target translation service, a translation control request is sent to the media capability platform, so that the media capability platform updates a processing operation based on the translation control request.

[0084] In another possible implementation, in step 303, before sending the translation service indication information to the media capability platform indicated by the media capability platform control address, the step further includes:

[0085] When it is determined that the target terminal does not have the translation service enabled, the call record of the target terminal is enabled.

[0086] In the embodiment of this application, Figure 6 As shown, the method for processing translation services during a video call may further include the following steps:

[0087] Step 601: During a call between a first terminal UE A and a second terminal UE B, the application server AS completes service pre-triggering according to service logic. For example, it sends a system-level setting request to the operation management platform, and sends the system-level setting data issued by the operation management platform based on the system-level setting request to the NCP, so that the NCP controls the UMF to play voice prompts within a specified time according to the system-level setting data; and sends a service-level call event subscription and a user-level call event subscription to the NCP to subscribe to call request events or call answer events of the target terminal.

[0088] Steps 602-605: The user presses a DTMF key on the target terminal to activate the voice transcription or subtitle translation function. The UMF detects this and reports it to the NCP. The NCP reports this command to the application server (AS) via a number collection result notification based on the DTMF command configuration file.

[0089] The application server (AS) determines the current business status and business change request information, and triggers different processing flows, as follows:

[0090] Steps 606a.1-606a.2: The application server AS determines that the user is currently using speech-to-text and needs to switch to translation, or that the user is currently using translation and needs to switch to speech-to-text. The application server AS then initiates a translation control request to the media capability platform to instruct the media capability platform to update the processing operation, for example, to change to translation or speech-to-text.

[0091] Steps 606b.1-606b.7: The application server AS determines that the user is not currently using speech-to-text or translation, and then the application server AS starts the speech-to-text / translation function-related business processing flow and opens the call record;

[0092] Among them, when the user operates the operation of switching the video call to the audio call during the call, the application server AS will stop the service and stop the call record, such as Figure 7 As shown, the steps are as follows:

[0093] Steps 701-702: During the translation service, the user initiates a video-to-audio switch, and the VoLTE AS receives a Re-INVITE.

[0094] Steps 703-704: The VoLTE AS reports the video-to-audio call event to the NCP, and the NCP reports the video-to-audio call event to the application server AS.

[0095] Steps 705-706: The video switches to the audio scene. The call event control sent by the application server AS must carry an ANCHOR indication to indicate that the audio continues to be anchored.

[0096] Step 707: The VoLTE AS updates the UMF audio and video media resources during this renegotiation and instructs the UMF to release the video media resources.

[0097] Steps 708-709: The VoLTE AS reports the re-negotiated media resource flow to the NCP, and the NCP reports it to the application server AS.

[0098] Steps 710 to 713: The application server AS instructs the UMF to stop media stream replication through the NCP, and instructs the media capability platform to stop the translation / transcription function and stop recording call records.

[0099] Figure 8 Another flow chart of a method for processing a translation service during a video call provided by an embodiment of the present application is shown. The method for processing a translation service during a video call is applied to a target terminal, which includes a first terminal and a second terminal. As shown in the figure, the method 800 for processing a translation service during a video call may include the following steps:

[0100] Step 801: Initiate a video call and establish a data channel with the unified media processing network element UMF;

[0101] Step 802: Obtain and download a translation application from the unified media processing network element UMF through the data channel; the translation application is embedded with the translation service subscription information corresponding to the target terminal;

[0102] Step 803: Acquire an interactive operation based on the translation application, and if it is determined according to the translation service subscription information that the translation service corresponding to the interactive operation is available, send the interactive operation to the UMF;

[0103] Step 804: Receive subtitle data sent by the UMF, synthesize the subtitle data with the media stream corresponding to the interactive operation, and display the synthesized video data; wherein the subtitle data is sent to the UMF by the media capability platform based on the received translation service indication information.

[0104] In an exemplary embodiment, Figure 9 As shown, taking the target terminal as terminal A as an example, the above method may include the following steps.

[0105] Step 910: Terminal A initiates a video call and establishes data channels BDC 0 and BDC 100 with the UMF. Here, BDC0 (Bearer Data Channel 0) is a basic channel used to carry voice or data and is used to initially establish a connection. BDC100 (Bearer Data Channel 100) is a channel used to carry specific services or data during the call, involving higher bandwidth or specific business requirements.

[0106] Step 920: Obtain and download the translation application from the UMF through the data channel, specifically including the following steps:

[0107] Steps 921-923: Terminal A requests the VoNR+ media plane to download the bootstrap program through BDC 0. The VoNR+ media plane obtains the bootstrap program from the VoNR+ capability network element and responds to the request through BDC 0.

[0108] Steps 924-926: Terminal A requests the VoNR+ media plane to download the translation application list through BDC 0. The VoNR+ media plane sends the request to the VoNR+ capability network element. The VoNR+ capability network element generates the mini-program list in real time based on user attributes and responds to the request through BDC 0.

[0109] Steps 931-934: Terminal A requests the VoNR+ media interface to download a translation application with the autoload attribute, for example, mini-program x, based on the mini-program list. Autoload refers to a mechanism for automatically loading translation service subscription information.

[0110] Step 940: The called terminal B picks up the phone and the call is established.

[0111] Steps 951-954: When the user clicks any non-autoload applet during a call, for example, the user clicks applet Y, terminal A will automatically request the VoNR+ media interface to download the translation application.

[0112] Further, if Figure 10 As shown, the process of starting a translation service may include the following steps:

[0113] Step 1001: Terminal A establishes a call with terminal B and completes downloading of a translation application;

[0114] Step 1002-Step 1003: After the user clicks on the translation application, the terminal completes downloading the translation application and starts the translation application;

[0115] Steps 1004-1014: The translation application triggers the establishment of an ADC channel through the new call software development kit (SDK). Terminal A negotiates with the VoNR+ media plane to complete the ADC establishment and complete the audio and video media anchor renegotiation.

[0116] Steps 1015-1019: The translation application requests the VoNR+ media plane through the new call SDK to obtain the current user subscription relationship, where the user subscription relationship includes multiple translation service subscription information. The VoNR+ media plane obtains the user's current subscription relationship from the application server AS and sends the obtained user's current subscription relationship information to the new call SDK through ADC1, and displays it in the translation application.

[0117] Steps 1021-1022: The application server (AS) checks the user's subscription status. If the user has not purchased the service, it initiates a subscription / billing confirmation. Specifically, in step 1020, the user clicks on the speech transcription / subtitle translation function. In step 1021, based on the data returned in step 1019, if it is determined that the user has not purchased the translation service, it enters the translation service subscription / billing page and executes steps 1022-1025.

[0118] Steps 1022-1025: After the user purchases the service on the service subscription / billing interface, the New Call SDK sends the user subscription / billing request to the VoNR+ media plane via the ADC. The VoNR+ media plane then notifies the application server (AS), triggering the service activation / billing process.

[0119] Step 1026: Based on the data returned in step 1019, if it is determined that the user has subscribed to the translation service, that is, the translation service corresponding to the interactive operation is available, then the speech transcription / subtitle translation service is started;

[0120] Steps 1027-1028: The translation application requests the new call ADK to call ADC1 to send a related request. After receiving the request, the new call SDK sends a command to start voice transcription / subtitle translation to the VoNR+ media plane. After receiving the command, the VoNR+ media plane reports it to the application server AS (built into the VoNR+ capability network element).

[0121] Step 1029: Determine whether a voice prompt switch is set. If yes, send a voice prompt to terminal B.

[0122] Steps 1030-1032: The application server (AS) initiates a media stream replication request. The VoNR+ media plane negotiates with the VoNR+ capability network to complete the media stream replication. Based on the media replication result, the VoNR+ capability network element returns information such as the media stream control address URL1 to the application server (AS). If the application server (AS) determines that the user media stream has not yet been replicated to the media capability platform, it executes the media stream replication process. If it has already been replicated to the media capability platform, it directly executes step 1033.

[0123] Step 1033: The application server AS initiates a translation control request to URL1 (media capability platform), carrying the subtitle receiving URL2, where URL2 is the DcStream resource URL information generated in step 1006. It instructs the media capability platform to perform speech-to-text or translation on the copied media stream and send the processing result to URL2.

[0124] Step 1034: After receiving a successful response from the media capability platform, the application server AS begins billing and records detailed services such as speech transcription or subtitle translation. If the application server AS determines that the user is currently using speech-to-text / voice assistant and needs to switch to translation, or if the user is currently using translation / voice assistant and needs to switch to speech-to-text, the application server AS stops recording the call log for the previous service function and starts recording the call log for the switching function.

[0125] Step 1035: Prompt the user that the voice transcription / subtitle translation command is successful;

[0126] Steps 1036-1037: The application server AS instructs execution of the target translation service and returns the translation result to the VoNR+ media plane address URL2 (VoNR+ media plane) through the UMF;

[0127] Step 1038: The VoNR+ media plane sends subtitle data to the new call SDK of terminal A through ADC2.

[0128] Step 1039: The new call SDK synthesizes the received subtitle data and the media stream corresponding to the interactive operation to obtain synthesized video data, and displays the synthesized video data on terminal A.

[0129] An embodiment of the present application provides a method for processing translation services during a video call. The target terminal initiates a video call and establishes a data channel with a unified media processing network element (UMF). The target terminal obtains and downloads a translation application from the unified media processing network element (UMF) via the data channel. The target terminal obtains an interactive operation based on the translation application and, if the translation service corresponding to the interactive operation is available based on the translation service subscription information, sends the interactive operation to the UMF. The target terminal receives subtitle data sent by the UMF, synthesizes the subtitle data with the media stream corresponding to the interactive operation, and displays the synthesized video data. This method can achieve efficient and reliable media stream translation services and improve the processing efficiency of audio stream translation services.

[0130] Figure 11 The schematic diagram of the structure of the application server provided in the embodiment of the present application is shown. The application server can implement the following Figure 3 In the embodiment shown, all or part of the content, as shown in the figure, the application server 1100 includes:

[0131] The number acquisition module 1110 is configured to determine whether a call link is established between the first terminal and the second terminal, and to acquire the number acquisition result of the target terminal; wherein the target terminal includes the first terminal and the second terminal;

[0132] The address acquisition module 1120 is configured to acquire an audio stream receiving address, a subtitle receiving address, and a media capability platform control address when it is determined based on the number collection result that the target terminal has enabled the target translation service;

[0133] The information sending module 1130 is used to send translation service indication information to the media capability platform indicated by the media capability platform control address, wherein the translation service indication information is used to instruct the media capability platform to obtain the audio copy stream based on the audio stream receiving address, and perform processing corresponding to the target translation service on the audio copy stream, and send the subtitle data obtained after processing to the target terminal corresponding to the subtitle receiving address.

[0134] In one possible implementation, the number acquisition module 1110, when determining to establish a call link between the first terminal and the second terminal, is specifically configured to:

[0135] Obtain a call request event of the first terminal and a call answer event of the second terminal sent by a new call service control platform NCP, and send a service control message to the NCP; wherein the service control message is used to instruct the NCP to anchor the audio and video media stream to the unified media processing network element UMF;

[0136] According to the audio and video anchoring control result returned by the NCP based on the service control message, it is determined that a call link is established between the first terminal and the second terminal.

[0137] In a possible implementation, the number acquisition module 1110 is further configured to:

[0138] Send a playback request to the new call service control platform NCP, which controls the UMF to play the voice prompt within a specified time and return the playback result after the voice prompt is played;

[0139] Obtain the announcement result returned by the NCP based on the announcement request.

[0140] In one possible implementation, the number acquisition module 1110 is further configured to:

[0141] Send a system-level setting request to the operation management platform, and send the system-level setting data issued by the operation management platform based on the system-level setting request to the NCP, so that the NCP controls the UMF to play the voice prompt within the specified time according to the system-level setting data.

[0142] In one possible implementation, the address acquisition module 1120, when used to acquire the audio stream receiving address, the subtitle receiving address, and the media capability platform control address, is specifically configured to:

[0143] Send a media stream translation request to the new call service control platform NCP, and obtain the audio stream receiving address, subtitle receiving address and media capability platform control address returned by the NCP based on the media stream translation request.

[0144] In one possible implementation, the address acquisition module 1120, when used to send a media stream translation request to the NCP and obtain the audio stream receiving address, subtitle receiving address, and media capability platform control address returned by the NCP based on the media stream translation request, is specifically configured to:

[0145] Send a media stream replication request to the NCP, where the media stream replication request is used to instruct the NCP to determine the media capability platform control address and notify the UMF to perform audio stream replication to obtain the audio stream receiving address corresponding to the UMF;

[0146] Obtaining the media capability platform control address and the audio stream receiving address returned by the NCP based on the media stream copy request;

[0147] Sending a subtitle stream synthesis request to the NCP, where the subtitle stream synthesis request is used to instruct the NCP to apply for subtitle stream synthesis resources from the UMF and obtain the subtitle receiving address returned by the UMF;

[0148] Acquire the subtitle receiving address returned by the NCP based on the subtitle stream synthesis request.

[0149] In a possible implementation, the information sending module 1130, before sending the translation service indication information to the media capability platform indicated by the media capability platform control address, is further configured to:

[0150] When it is determined that the translation service currently used by the target terminal cannot match the target translation service, a translation control request is sent to the media capability platform, so that the media capability platform updates a processing operation based on the translation control request.

[0151] In a possible implementation, the information sending module 1130, before sending the translation service indication information to the media capability platform indicated by the media capability platform control address, is further configured to:

[0152] When it is determined that the target terminal does not have the translation service enabled, the call record of the target terminal is enabled.

[0153] In a possible implementation, the information sending module 1130, when used to send the processed subtitle data to the target terminal corresponding to the subtitle receiving address, is specifically configured to:

[0154] The subtitle data obtained after processing is sent to the unified media processing network element UMF indicated by the subtitle receiving address, and the subtitle data is sent to the target terminal through the UMF.

[0155] An embodiment of the present application provides an application server comprising: a number acquisition module, an address acquisition module, and an information sending module; the number acquisition module determines that a call link has been established between a first terminal and a second terminal, and obtains a number acquisition result for a target terminal; the address acquisition module, upon determining that the target terminal has enabled a target translation service based on the number acquisition result, obtains an audio stream receiving address, a subtitle receiving address, and a media capability platform control address; and the information sending module sends translation service indication information to the media capability platform indicated by the media capability platform control address. Thus, through standardized interfaces and unified business logic, an efficient and reliable media stream translation service can be implemented, the processing efficiency of the audio stream translation service can be improved, and the service can be more scalable and maintainable.

[0156] Figure 12 The terminal provided in the embodiment of the present application is shown in the structural diagram. The terminal can implement the following Figure 8In the embodiment shown, all or part of the contents, as shown in the figure, the terminal 1200 includes:

[0157] The channel establishment module 1210 is used to initiate a video call and establish a data channel with the unified media processing network element UMF;

[0158] The program download module 1220 is configured to obtain and download a translation application from the unified media processing network element UMF via the data channel; the translation application is embedded with the translation service subscription information corresponding to the target terminal;

[0159] An operation sending module 1230 is configured to obtain an interactive operation based on the translation application and, if the translation service corresponding to the interactive operation is determined to be available based on the translation service subscription information, send the interactive operation to the UMF. A data receiving module is configured to receive subtitle data sent by the UMF, synthesize the subtitle data with the media stream corresponding to the interactive operation, and display the synthesized video data. The subtitle data is sent to the UMF by the media capability platform based on the received translation service indication information.

[0160] The embodiment of the present application provides a terminal, including a channel establishment module, a program download module, and an operation sending module; the channel establishment module initiates a video call and establishes a data channel with a unified media processing network element (UMF); the program download module obtains and downloads a translation application from the unified media processing network element (UMF) through the data channel; the operation sending module obtains an interactive operation based on the translation application, and when it is determined that the translation service corresponding to the interactive operation is available according to the translation service subscription information, the interactive operation is sent to the UMF; the data receiving module receives the subtitle data sent by the UMF, synthesizes the subtitle data with the media stream corresponding to the interactive operation, and displays the synthesized video data. In this way, an efficient and reliable media stream translation service can be implemented, and the processing efficiency of the audio stream translation service can be improved.

[0161] Figure 13 A schematic diagram of the hardware structure of an electronic device that implements the embodiments of the present application is shown. Referring to the figure, at the hardware level, the electronic device 1300 includes a processor 1310, and optionally, an internal bus 1320, a network interface 1330, and a memory 1340. Among them, the memory 1340 may include a memory 1341, such as a high-speed random access memory (RAM), and may also include a non-volatile memory 1342 (non-volatile memory), such as at least one disk storage device. Of course, the electronic device 1300 may also include hardware required for other services.

[0162] The processor 1310, the network interface 1330, and the memory can be interconnected via an internal bus 1320. The internal bus 1320 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Such buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, only one bidirectional arrow is used in this figure, but this does not imply that there is only one bus or only one type of bus.

[0163] The memory 1340 stores programs. Specifically, the programs may include program codes, which include computer operating instructions. The memory 1340 may include a memory 1341 and a non-volatile memory 1342, and provides instructions and data to the processor 1310.

[0164] The processor 1310 reads the corresponding computer program from the non-volatile memory 1342 into the memory and then runs it, forming a device for locating the target user at the logical level. The processor 1310 executes the program stored in the memory and specifically performs the following: Figure 3 or Figure 8 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0165] The above application Figure 3 or Figure 8The methods disclosed in the illustrated embodiments can be applied to or implemented by the processor 1310. The processor 1310 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1310. The above-mentioned processor 1310 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0166] The computer device can also execute the methods described in the above method embodiments and realize the functions and beneficial effects of the methods described in the above method embodiments, which will not be repeated here.

[0167] Of course, in addition to software implementation, the electronic device 1300 of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0168] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, which, when executed by an electronic device including a plurality of application programs, enables the electronic device to execute Figure 3 or Figure 8 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0169] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0170] Furthermore, an embodiment of the present application provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer, the following process is implemented: Figure 3 or Figure 8 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0171] The embodiments of the present application can be applied to various electronic device collaboration or interconnection scenarios, including: collaboration and interconnection between mobile phones and laptops / tablets; collaboration and interconnection between mobile terminals and smart TVs / displays; collaboration and interconnection between mobile phones or tablets and in-car entertainment systems; collaboration and interconnection between mobile terminals and smart conference systems, etc., thereby meeting the diverse needs of users in scenarios such as smart homes, smart offices, and smart travel.

[0172] In short, the above description is only a preferred embodiment of the present application and does not limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0173] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0174] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0176] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A method for processing translation services during a video call, characterized in that: Applied to the application server AS, including: Determine that a call link is established between the first terminal and the second terminal, and obtain a number collection result of a target terminal; wherein the target terminal includes the first terminal and the second terminal; In the case where it is determined based on the number collection result that the target terminal starts the target translation service, obtaining an audio stream receiving address, a subtitle receiving address, and a media capability platform control address; Send translation service indication information to the media capability platform indicated by the media capability platform control address, wherein the translation service indication information is used to instruct the media capability platform to obtain an audio copy stream based on the audio stream receiving address, and perform processing corresponding to the target translation service on the audio copy stream, and send the processed subtitle data to the target terminal corresponding to the subtitle receiving address.

2. The method according to claim 1, characterized in that The determining to establish a call link between the first terminal and the second terminal includes: Obtain a call request event of the first terminal and a call answer event of the second terminal sent by a new call service control platform NCP, and send a service control message to the NCP; wherein the service control message is used to instruct the NCP to anchor the audio and video media stream to the unified media processing network element UMF; According to the audio and video anchoring control result returned by the NCP based on the service control message, it is determined that a call link is established between the first terminal and the second terminal.

3. The method according to claim 1, characterized in that Before obtaining the number receiving result of the target terminal, the method further includes: Send a playback request to the new call service control platform NCP, and the NCP controls the UMF to play the voice prompt within a specified time and return the playback result after the voice prompt is played; The announcement result returned by the NCP based on the announcement request is obtained.

4. The method according to claim 3, characterized in that Before sending the playback request to the new call service control platform NCP, the method further includes: Send a system-level setting request to the operation management platform, and send the system-level setting data issued by the operation management platform based on the system-level setting request to the NCP, so that the NCP controls the UMF to play the voice prompt within a specified time according to the system-level setting data.

5. The method according to claim 1, characterized in that: The step of obtaining the audio stream receiving address, the subtitle receiving address, and the media capability platform control address includes: A media stream translation request is sent to a new call service control platform NCP, and an audio stream receiving address, a subtitle receiving address, and a media capability platform control address returned by the NCP based on the media stream translation request are obtained.

6. The method according to claim 5, characterized in that The sending of the media stream translation request to the NCP, and obtaining the audio stream receiving address, the subtitle receiving address, and the media capability platform control address returned by the NCP based on the media stream translation request, include: Send a media stream replication request to the NCP, where the media stream replication request is used to instruct the NCP to determine the media capability platform control address and notify the UMF to perform audio stream replication to obtain the audio stream receiving address corresponding to the UMF; Acquire the media capability platform control address and the audio stream receiving address returned by the NCP based on the media stream replication request; Sending a subtitle stream synthesis request to the NCP, wherein the subtitle stream synthesis request is used to instruct the NCP to apply for subtitle stream synthesis resources from the UMF and obtain the subtitle receiving address returned by the UMF; The subtitle receiving address returned by the NCP based on the subtitle stream synthesis request is obtained.

7. The method according to claim 1, characterized in that Before sending the translation service indication information to the media capability platform indicated by the media capability platform control address, the method further includes: When it is determined that the translation service currently used by the target terminal cannot match the target translation service, a translation control request is sent to the media capability platform, so that the media capability platform updates the processing operation based on the translation control request.

8. The method according to claim 1, characterized in that Before sending the translation service indication information to the media capability platform indicated by the media capability platform control address, the method further includes: When it is determined that the target terminal does not have the translation service enabled, the call record of the target terminal is enabled.

9. The method according to any one of claims 1 to 8, characterized in that: The step of sending the processed subtitle data to a target terminal corresponding to the subtitle receiving address includes: The subtitle data obtained after processing is sent to the unified media processing network element UMF indicated by the subtitle receiving address, and the subtitle data is sent to the target terminal through the UMF.

10. A method for processing translation services during a video call, characterized in that: Applied to target terminals, including: Initiate a video call and establish a data channel with the unified media processing network element UMF; Obtaining and downloading a translation application from the unified media processing network element UMF through the data channel; the translation application is embedded with the translation service subscription information corresponding to the target terminal; Acquire an interactive operation based on the translation application, and send the interactive operation to the UMF when determining that the translation service corresponding to the interactive operation is available according to the translation service subscription information; receiving subtitle data sent by the UMF, synthesizing the subtitle data with a media stream corresponding to the interactive operation, and displaying the synthesized video data; The subtitle data is sent to the UMF by the media capability platform based on the received translation service indication information.

Images