Methods, systems, electronic devices and storage media for displaying translated subtitles during phone calls

Abstract

This invention discloses a method, system, electronic device, and storage medium for displaying translated subtitles during phone calls. The method includes: triggering the network side to complete audio and video media anchoring and establishing an ADC channel between a first calling terminal and the network side; responding to a translation start command from the first calling terminal, initiating a media stream copying request to the data channel signaling function network element via an intelligent translation application server, causing the media function network element to transmit the uplink audio stream from the second calling terminal to the media processing function network element; translating the uplink audio stream to obtain a subtitle stream via the media processing function network element and returning the subtitle stream to the media function network element; and combining the subtitle stream with the downlink video stream to obtain a subtitle video stream for display. This invention improves call efficiency and the accuracy of information acquisition and can be widely applied in the field of communication technology.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method, system, electronic device, and storage medium for displaying translated subtitles during phone calls. Background Technology

[0002] With the widespread adoption and application of 5G networks, and the development of 5G-enhanced real-time communication technology, mobile communication has become more efficient and convenient, improving real-time call quality and user experience. Accelerated globalization has increased the demand for cross-language communication, but using different languages ​​in calls can lead to communication difficulties. Traditionally, playing translated audio after speech recognition alters the original call audio, causing information errors and affecting call efficiency and the accuracy of information acquisition.

[0003] Terminology Explanation:

[0004] BDC (Bootstrap Data Channel): Established between the terminal and the IMS network, it is used by the terminal to obtain a list of data channel applications from the IMS network and download the required data channel applications.

[0005] ADC (Application Data Channel): Established between the terminal and the IMS network or between the terminal, it is used to transmit application data of the application between the two ends. Summary of the Invention

[0006] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0007] Therefore, one objective of this invention is to provide a method for displaying translated subtitles during phone calls, which improves call efficiency and the accuracy of information acquisition.

[0008] Another objective of this invention is to provide a call translation subtitle display system.

[0009] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include:

[0010] On one hand, embodiments of the present invention provide a method for displaying translated subtitles during phone calls, comprising the following steps:

[0011] The network side is triggered to complete the audio and video media anchoring and establish the ADC channel between the first calling terminal and the network side.

[0012] In response to the translation start command from the first call terminal, a media stream copying request is initiated to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element.

[0013] The media processing function network element translates the uplink audio stream to obtain a subtitle stream, and then returns the subtitle stream to the media function network element.

[0014] The subtitle stream is sent to the first call terminal via the media function network element, so that the first call terminal combines the subtitle stream with the downlink video stream to obtain a subtitle video stream and displays it. Alternatively, the subtitle stream and downlink video stream are combined via the media function network element to obtain a subtitle video stream, and the subtitle video stream is sent to the first call terminal for display.

[0015] Furthermore, in one embodiment of the present invention, the triggering network side completes audio and video media anchoring and establishes an ADC channel between the first calling terminal and the network side, which specifically includes:

[0016] The intelligent translation mini-program triggers the 5G enhanced call SDK to request the establishment of the ADC channel from the core network side, so that the core network side sends a media change call event notification to the intelligent translation application server through the data channel signaling function network element.

[0017] The intelligent translation application server calls the call control northbound interface provided by the data channel signaling function network element to trigger the network side to perform audio and video media anchor renegotiation and complete the establishment of the ADC channel.

[0018] The ADC channel includes a first ADC channel for service control and a second ADC channel for subtitle reception.

[0019] Furthermore, in one embodiment of the present invention, the call translation subtitle display method further includes the following steps:

[0020] The intelligent translation mini-program triggers the 5G enhanced call SDK to request the media function network element to obtain the user subscription relationship of the first call terminal, so that the media function network element obtains the user subscription relationship from the intelligent translation application server and returns the user subscription relationship to the intelligent translation mini-program;

[0021] The intelligent translation mini-program determines whether the first calling terminal has subscribed to the subtitle translation service based on the user's subscription relationship. If the first calling terminal has not subscribed to the subtitle translation service, the intelligent translation mini-program guides the user to activate the service.

[0022] Furthermore, in one embodiment of the present invention, the step of responding to the translation start command from the first call terminal by initiating a media stream copy request to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element, specifically includes:

[0023] The intelligent translation mini-program triggers the 5G enhanced call SDK to call the first ADC channel to send the start translation command to the media function network element, so that the media function network element reports the start translation command to the intelligent translation application server;

[0024] The intelligent translation application server initiates a media stream copying request to the data channel signaling function network element, enabling the data channel signaling function network element and the media function network element to negotiate and complete the media stream copying, and then the media function network element transmits the copied uplink audio stream to the media processing function network element.

[0025] The data channel signaling function network element returns the media stream control address to the intelligent translation application server.

[0026] Furthermore, in one embodiment of the present invention, the step of translating the uplink audio stream into a subtitle stream through the media processing function network element and returning the subtitle stream to the media function network element specifically includes:

[0027] The intelligent translation application server initiates an intelligent translation control request to the media processing function network element based on the media stream control address, and the intelligent translation control request carries the subtitle receiving address.

[0028] The media processing function network element performs speech recognition and translation operations on the uplink audio stream to obtain the subtitle stream, and returns the subtitle stream to the media function network element according to the subtitle receiving address.

[0029] Furthermore, in one embodiment of the present invention, the synthesis of the subtitle stream and the downlink video stream specifically includes:

[0030] The subtitle stream is parsed to obtain multiple subtitle texts and their corresponding first timestamps;

[0031] The downlink video stream is decoded to obtain multiple video frames and corresponding second timestamps;

[0032] The subtitle text and the video frame are matched based on the first timestamp and the second timestamp to obtain the matched target video frame and target subtitle text;

[0033] A subtitle image is generated based on a preset subtitle style template and the target subtitle text, and the subtitle image is superimposed on the target video frame to obtain a subtitle video frame;

[0034] The subtitle video frames are video encoded to obtain the subtitle video stream.

[0035] Furthermore, in one embodiment of the present invention, the step of matching the subtitle text and the video frame according to the first timestamp and the second timestamp to obtain the matched target video frame and target subtitle text specifically includes:

[0036] Multiple candidate video frames corresponding to the target subtitle text are determined based on the first timestamp and the second timestamp;

[0037] The candidate video frames are subjected to speech recognition to obtain text content, and the cosine similarity between the text content and the target subtitle text is calculated.

[0038] When the cosine similarity is greater than or equal to a preset first threshold, the candidate video frame is determined to be the target video frame.

[0039] On the other hand, embodiments of the present invention provide a call translation subtitle display system, including:

[0040] The channel establishment module is used to trigger the network side to complete the audio and video media anchoring and establish the ADC channel between the first call terminal and the network side.

[0041] The audio stream copying module is used to respond to the translation start command of the first call terminal and initiate a media stream copying request to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element.

[0042] The audio stream translation module is used to translate the uplink audio stream into a subtitle stream through the media processing function network element, and return the subtitle stream to the media function network element;

[0043] The subtitle video synthesis module is used to send the subtitle stream to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream and the downlink video stream to obtain a subtitle video stream and displays it; or, it synthesizes the subtitle stream and the downlink video stream through the media function network element to obtain a subtitle video stream and sends the subtitle video stream to the first call terminal for display.

[0044] On the other hand, embodiments of the present invention provide an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the call translation subtitle display method described above.

[0045] On the other hand, embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the call translation subtitle display method as described above.

[0046] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention:

[0047] In this embodiment of the invention, the network side is triggered to complete audio and video media anchoring and establish an ADC channel between the first call terminal and the network side. In response to the translation start command from the first call terminal, the intelligent translation application server initiates a media stream copying request to the data channel signaling function network element. This causes the media function network element to transmit the uplink audio stream of the second call terminal to the media processing function network element. The media processing function network element translates the uplink audio stream to obtain a subtitle stream and returns the subtitle stream to the media function network element. The media function network element then sends the subtitle stream to the first call terminal, allowing the first call terminal to synthesize the subtitle stream and the downlink video stream to obtain a subtitle video stream for display. Alternatively, the media function network element can synthesize the subtitle stream and the downlink video stream to obtain a subtitle video stream, which is then sent to the first call terminal for display. This invention superimposes translated subtitles during video calls, achieving translation by displaying subtitles on the terminal screen without affecting the original call audio, reducing information errors, and improving call efficiency and the accuracy of information acquisition. Based on the operator's existing IMS core network, system enhancements and service function enhancements are made to display translated subtitles during video calls on the user terminal without the need for additional translation software installation, simplifying the user operation process and improving the user experience. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1A flowchart illustrating one step of the call translation subtitle display method provided in this embodiment of the invention;

[0050] Figure 2 This is a schematic diagram illustrating an application scenario of the call translation subtitle display method provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the instructions for the call translation subtitle display method provided in an embodiment of the present invention;

[0052] Figure 4 A flowchart of step S101 provided in an embodiment of the present invention;

[0053] Figure 5 A flowchart of step S102 provided in an embodiment of the present invention;

[0054] Figure 6 A flowchart of step S103 provided in an embodiment of the present invention;

[0055] Figure 7 A flowchart of step S104 provided in an embodiment of the present invention;

[0056] Figure 8 A flowchart of step S1043 provided in an embodiment of the present invention;

[0057] Figure 9 This is a schematic diagram of the call translation subtitle display system provided in an embodiment of the present invention;

[0058] Figure 10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention;

[0059] Figure 11 This is a schematic diagram of the structure of the storage medium provided in an embodiment of the present invention. Detailed Implementation

[0060] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are only set for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0061] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] The call translation subtitle display method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, set-top box, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the call translation subtitle display method, but is not limited to the above forms.

[0063] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0064] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0065] First, the application scenarios of the embodiments of the present invention will be described. For example... Figure 2 The diagram illustrates an application scenario of the call translation subtitle display method provided in this embodiment of the invention, including the core network side, the enhanced call network side, the terminal side, and the operating environment side. The relevant descriptions are as follows:

[0066] 1) Core Network Side: Existing communication network elements of the operator. EPS / 5GS: Used to carry data and call control signals. P-CSCF / BAC: Responsible for establishing and maintaining call sessions between user equipment and the IMS network. I / S-CSCF: Used to receive and process SIP call requests and perform call control functions. MMTel AS: Used to provide application services for multimedia telephony services (such as VoLTE, Video over LTE, etc.). HSS / UDM: Central repository for user data, supporting functions such as authentication, subscription management, mobility management, and session establishment. ENUM / DNS: Used to map telephone numbers to Internet Resource Identifiers.

[0067] 2) Data Channel Signaling Function (DCSF): An application supporting access management for 5G enhanced real-time communication services. It possesses multiple functions, including providing atomic operation capabilities of Media Functions (MF) to Service Application Services (AS), subscribing to call events, integrating subscription events to the MMTel AS, and transmitting call-related information to the Service AS. Furthermore, DCSF also supports storing user information in the HSS / UDM, selecting MPF addresses, storing DC application clients (DCAR function), and controlling the processing of the Bootstrap DC (BDC), among other functions.

[0068] 3) Media Function (MF): A multi-functional media processing application. It can receive and control commands from MMTelAS and DCSF network elements, including the creation, updating, and deletion of media resources, media stream copying, basic audio and video capabilities, and media compositing. Furthermore, MF supports the downloading and caching of media resources and facilitates data channel interaction with DCSF network elements.

[0069] 4) Media Processing Function (MPF): Provides richer media processing capabilities, supporting speech recognition to text, translation, and other AI capabilities.

[0070] 5) Intelligent Translation Application Server (Service AS): A key component responsible for implementing business logic and interacting with DCSF network elements. During registration, the Service AS must carry a device-level subscription instruction to inform the DCSF of the call events that need to be reported, and support instructing the media capability platform to perform the necessary media processing. The Service AS provides the business logic and application interaction content of the DC application server. The Service AS can effectively manage business processes and collaborate with other systems to complete the intelligent translation business logic control.

[0071] 6) Smart Translation Mini Program: Provides an H5 front-end operation interface to realize intelligent translation start / stop control and style configuration, etc.

[0072] 7) 5G Enhanced Call SDK: Provides capabilities such as application data channel creation and closure, DC data transmission, and terminal rendering, serving as a bridge for data interaction between the smart translation mini-program and the enhanced call network side.

[0073] In this embodiment of the invention, after user terminal A and user terminal B establish a call, the translation function is initiated through a smart translation mini-program. The mini-program establishes a data channel through the 5G Enhanced Call SDK (hereinafter referred to as SDK) and obtains the user's subscription information to confirm whether the user has subscribed to the translation service. If not subscribed, the user is guided through the subscription and billing steps; if subscribed, the translation service is initiated. The smart translation application server (hereinafter referred to as the service AS) requests media stream copying and initiates a translation request to the Media Capability Platform (hereinafter referred to as MPF), sending the processing result to the terminal for display. After the translation is completed, the translated subtitles are synthesized and displayed on the terminal, providing the user with a convenient cross-language call experience.

[0074] like Figure 1 The diagram shown is a flowchart of one step of the call translation subtitle display method provided by an embodiment of the present invention, as follows: Figure 3 The diagram shown is an instruction illustration of the call translation subtitle display method provided in an embodiment of the present invention. (Refer to...) Figure 1 and 3 This invention provides a method for displaying translated subtitles during phone calls, specifically including the following steps:

[0075] S101, Trigger the network side to complete audio and video media anchoring and establish the ADC channel between the first call terminal and the network side.

[0076] like Figure 4 The diagram shown is a flowchart of step S101 provided in an embodiment of the present invention. (Refer to...) Figure 4 As an optional implementation, the network side is triggered to complete audio and video media anchoring and establish an ADC channel between the first calling terminal and the network side, which specifically includes:

[0077] S1011. The 5G Enhanced Call SDK is triggered by the intelligent translation mini-program to request the establishment of the ADC channel from the core network side, so that the core network side sends a media change call event notification to the intelligent translation application server through the data channel signaling function network element.

[0078] S1012. The network side is triggered to perform audio and video media anchor renegotiation by calling the call control northbound interface provided by the data channel signaling function network element through the intelligent translation application server, and the establishment of the ADC channel is completed through negotiation.

[0079] The ADC channel includes a first ADC channel for service control and a second ADC channel for subtitle reception.

[0080] Specifically, the intelligent translation mini-program triggers the establishment of an ADC channel through the 5G enhanced calling SDK. After receiving the media change call event notification, the service AS calls the call control northbound interface provided by DCSF to trigger the network side to complete the audio and video media anchor renegotiation and negotiate the establishment of the ADC between terminal A and the network side. The intelligent translation service requires the establishment of two ADCs: ADC1 for service control and ADC2 for subtitle reception.

[0081] As a further optional implementation, the call translation subtitle display method also includes the following steps:

[0082] The intelligent translation mini-program triggers the 5G enhanced call SDK to request the media function network element to obtain the user subscription relationship of the first call terminal, so that the media function network element obtains the user subscription relationship from the intelligent translation application server and returns the user subscription relationship to the intelligent translation mini-program.

[0083] The intelligent translation mini-program determines whether the first calling terminal has subscribed to the subtitle translation service based on the user's subscription relationship. If the first calling terminal has not subscribed to the subtitle translation service, the intelligent translation mini-program guides the user to activate the service.

[0084] Specifically, the smart translation mini-program requests the current user subscription relationship from the MF via the SDK. The MF obtains the user's current subscription relationship from the smart translation AS and sends the obtained user subscription relationship information to the SDK via ADC1, which is then displayed within the smart translation mini-program. If the user has not subscribed, they are guided to complete the subscription and billing process; if they have subscribed, the translation function is activated.

[0085] S102. In response to the translation start command from the first call terminal, a media stream copying request is initiated to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element.

[0086] like Figure 5 The diagram shown is a flowchart of step S102 provided in an embodiment of the present invention. (Refer to...) Figure 5 As an optional implementation, in response to the translation initiation command from the first calling terminal, a media stream copying request is initiated to the data channel signaling function network element through the intelligent translation application server, causing the media function network element to transmit the uplink audio stream of the second calling terminal to the media processing function network element, specifically including:

[0087] S1021. The 5G enhanced call SDK is triggered by the intelligent translation mini-program to call the first ADC channel to send a start translation command to the media function network element, so that the media function network element reports the start translation command to the intelligent translation application server.

[0088] S1022. The intelligent translation application server initiates a media stream replication request to the data channel signaling function network element, so that the data channel signaling function network element and the media function network element negotiate to complete the media stream replication, and transmit the replicated uplink audio stream to the media processing function network element through the media function network element.

[0089] S1023. The media stream control address is returned to the intelligent translation application server through the data channel signaling function network element.

[0090] Specifically, the intelligent translation mini-program sends a translation start request to ADC1 via the request SDK. After receiving the request, the SDK sends a translation start command to MF. Upon receiving the command, MF reports it to the intelligent translation application server. The intelligent translation AS initiates a media stream copy request to DCSF. MF and DCSF negotiate to complete the media stream copy, copying the uplink audio stream of terminal B from MF to MPF. Based on the copy result, DCSF returns information such as the media stream control address URL1 to the service AS.

[0091] S103. The uplink audio stream is translated into a subtitle stream through the media processing function network element, and the subtitle stream is returned to the media function network element.

[0092] like Figure 6 The diagram shown is a flowchart of step S103 provided in an embodiment of the present invention. (Refer to...) Figure 6 As an optional implementation, the uplink audio stream is translated into a subtitle stream by a media processing function network element, and the subtitle stream is returned to the media function network element. Specifically, this includes:

[0093] S1031. The intelligent translation application server initiates an intelligent translation control request to the media processing function network element based on the media stream control address. The intelligent translation control request carries the subtitle receiving address.

[0094] S1032. The media processing function network element performs speech recognition and translation on the uplink audio stream to obtain the subtitle stream, and returns the subtitle stream to the media function network element according to the subtitle receiving address.

[0095] Specifically, the service AS initiates a smart translation control request to URL1 (MPF), instructing the MPF to perform speech recognition and translation operations. The request includes the subtitle receiving address URL2, which the MPF will use to send the translation result to the MF in the subsequent response. The service AS begins recording billing details. The service AS sends a successful translation initiation request via ADC1, which is transparently transmitted through intermediate network elements such as the MF and SDK, and reported to the smart translation mini-program. The mini-program then provides the user with a prompt about the operation result. The MPF performs speech recognition and translation on the uplink audio stream of terminal B and returns the translation result to the MF based on the subtitle receiving address URL2.

[0096] S104. The subtitle stream is sent to the first call terminal through the media function network element, so that the first call terminal combines the subtitle stream and the downlink video stream to obtain a subtitle video stream and displays it. Alternatively, the subtitle stream and the downlink video stream are combined through the media function network element to obtain a subtitle video stream, and the subtitle video stream is sent to the first call terminal for display.

[0097] Specifically, this invention provides the following two methods for subtitle rendering:

[0098] On the terminal side, the SDK renders the subtitle stream. MF uses ADC2 to send the translated subtitle stream to the SDK of terminal A. The SDK receives and synthesizes the subtitle stream on terminal A.

[0099] Network-side rendering: MF combines the subtitle stream content and subtitle template to synthesize the downlink video stream of terminal A, and then sends the synthesized subtitled video stream to terminal A for display.

[0100] This technology utilizes a cosine similarity algorithm (subtitle content matching algorithm) to match subtitle content and an Alpha synthesis algorithm (subtitle style synthesis algorithm) to synthesize subtitle styles. Then, an overlay and fusion algorithm (overlay algorithm) is used to overlay the synthesized subtitles onto the original video stream. Finally, H.264 video encoding is employed to encode and compress the processed video, ensuring efficient transmission of the subtitled video stream. This technique, encompassing subtitle content matching, subtitle style synthesis, and overlay algorithms, achieves efficient synthesis of subtitled videos, enhancing the user's audiovisual experience.

[0101] like Figure 7 The diagram shown is a flowchart of step S104 provided in an embodiment of the present invention. (Refer to...) Figure 7 As an optional implementation, the subtitle stream and the downlink video stream are further combined, which specifically includes:

[0102] S1041. Parse the subtitle stream to obtain multiple subtitle texts and their corresponding first timestamps;

[0103] S1042. Decode the downlink video stream to obtain multiple video frames and their corresponding second timestamps;

[0104] S1043. Match the subtitle text and video frame according to the first timestamp and the second timestamp to obtain the matched target video frame and target subtitle text;

[0105] S1044. Generate a subtitle image based on the preset subtitle style template and the target subtitle text, and overlay the subtitle image with the target video frame to obtain the subtitle video frame;

[0106] S1045. Perform video encoding on the subtitle video frames to obtain the subtitle video stream.

[0107] Specifically, the first step involves preprocessing the subtitle and video streams using FFmpeg. This includes parsing the subtitles to extract the text and corresponding timestamps, and decoding the video to break it down into a series of timestamped frames. Next, subtitle content matching is performed, aiming to match the text in the subtitle stream to the corresponding frames in the video stream. This is achieved by comparing the timestamps in the subtitle and video streams. Then, subtitle style synthesis is performed based on a style template. This step generates a subtitle image, where the style template defines the subtitle's position, font, color, size, background, and other attributes. First, an image containing the subtitle text is generated based on the style template. Then, an alpha synthesis algorithm is used to synthesize this image with a transparent background to generate the final subtitle image. The synthesized subtitle image is then superimposed onto the original video stream. This step is achieved using the OpenCV-Python image fusion function `cv2.addWeighted`, which takes the subtitle image, video frame, image weights, and an output scalar as input to complete the image synthesis. Finally, the video frame with added subtitles is re-encoded into a video stream and output. This step is accomplished using the H.264 video encoding algorithm and the `ffmpeg.output` function of FFmpeg.

[0108] like Figure 8 The diagram shown is a flowchart of step S1043 provided in an embodiment of the present invention. (Refer to...) Figure 8 As an optional implementation, the subtitle text and video frame are matched based on the first timestamp and the second timestamp to obtain the matched target video frame and target subtitle text, specifically including:

[0109] S10431. Determine multiple candidate video frames corresponding to the target subtitle text based on the first and second timestamps;

[0110] S10432. Perform speech recognition on candidate video frames to obtain text content, and calculate the cosine similarity between the text content and the target subtitle text.

[0111] S10433. When the cosine similarity is greater than or equal to the preset first threshold, the candidate video frame is determined as the target video frame.

[0112] Specifically, to further optimize subtitle content matching, a cosine similarity algorithm can be used. If the cosine similarity between the text in a frame of the video stream and the subtitle text in the subtitle stream is higher than a certain threshold, then this frame is considered to correspond to this subtitle.

[0113] In some optional embodiments, after user terminal A and user terminal B establish a call, the smart translation mini-program establishes a data channel through the SDK to obtain user subscription information and verify the translation service subscription status. If not subscribed, the user is guided to complete the subscription and billing process; if subscribed, the translation function is activated. The service AS sends a media stream copy request to the MPF and initiates a translation request. The MPF returns the translated subtitle stream to the MF. The MF determines whether the target terminal supports terminal-side rendering based on the call signaling parameters. For terminals that support rendering, the SDK completes the subtitle display; for terminals that do not support rendering, the MF completes the downlink video subtitle synthesis and sends the video stream with subtitles. This method effectively utilizes terminal resources and reduces the subtitle rendering burden on the network-side MF element, but requires communication terminal manufacturers to adapt the software to support terminal-side rendering.

[0114] In some optional embodiments, after user terminal A and user terminal B establish a call connection, the smart translation mini-program establishes a data channel through the SDK to obtain user subscription information and verify the translation service subscription status. If no subscription has been made, the user is guided to complete the subscription and billing process; if a subscription has been made, the translation function is activated. The service AS sends a media stream copy request to the MPF and initiates a translation request. The MPF returns the translated subtitle stream to the MF, and the MF completes the subtitle synthesis for the downlink video and sends the video stream with subtitles to the current terminal. This solution only has basic network-side subtitle rendering capabilities, and its implementation difficulty and cost are relatively low.

[0115] The method flow and implementation process of this invention have been described above. It is understood that, to ensure the stability of MF's basic media synthesis capabilities and to provide a more flexible access path for AI functions that may change frequently, this invention separates AI functions that are more likely to change, such as speech recognition to text and translation, from MF as an independent Media Processing Function (MPF) service. This design enables more convenient access and integration with third-party AI capabilities while ensuring the stable operation of MF's basic media synthesis capabilities. The MPF service will independently handle speech recognition to text, translation, and other functions, providing customers and partners with standard interfaces and open access specifications to ensure efficient collaboration and integration. MF remains focused on providing basic media processing capabilities, such as audio and video resource management, media stream replication, and basic capability support, maintaining its reliability and stability. Through this architectural design, this invention can better cope with changes in AI functions and iterative technological innovations, while providing more flexible and scalable solutions for business.

[0116] This invention superimposes translated subtitles during video calls, achieving translation by displaying subtitles on the terminal screen without affecting the original call audio, reducing information errors, and improving call efficiency and the accuracy of information acquisition. Based on the operator's existing IMS core network, system enhancements and service function enhancements are made to display translated subtitles during video calls on the user terminal without the need for additional translation software installation, simplifying the user operation process and improving the user experience.

[0117] like Figure 9 The diagram shown is a structural schematic of the call translation subtitle display system provided in an embodiment of the present invention. (Refer to...) Figure 9 This invention provides a call translation subtitle display system, comprising:

[0118] The channel establishment module is used to trigger the network side to complete the audio and video media anchoring and establish the ADC channel between the first call terminal and the network side.

[0119] The audio stream copying module is used to respond to the translation start command of the first call terminal and initiate a media stream copying request to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element.

[0120] The audio stream translation module is used to translate the uplink audio stream into a subtitle stream through the media processing function network element, and then return the subtitle stream to the media function network element.

[0121] The subtitle video synthesis module is used to send the subtitle stream to the first call terminal through the media function network element, so that the first call terminal can synthesize the subtitle stream and the downlink video stream to obtain the subtitle video stream and display it, or synthesize the subtitle stream and the downlink video stream through the media function network element to obtain the subtitle video stream and send the subtitle video stream to the first call terminal for display.

[0122] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0123] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned call translation subtitle display method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0124] like Figure 10 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of the present invention. (Refer to...) Figure 10 This invention provides an electronic device, comprising:

[0125] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0126] The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the call translation subtitle display method of the embodiments of the present invention.

[0127] Input / output interface 1003 is used to implement information input and output;

[0128] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0129] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0130] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0131] like Figure 11 The diagram shown is a structural schematic of the storage medium provided in an embodiment of the present invention. (Refer to...) Figure 11 The present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 1101, which can be executed by one or more processors to implement the above-described call translation subtitle display method.

[0132] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0133] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0134] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0135] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0136] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0138] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or, if necessary, processing in other suitable ways, and then stored in computer memory.

[0139] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0140] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0141] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0142] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for displaying translated subtitles during phone calls, characterized in that, Includes the following steps: The network side is triggered to complete the audio and video media anchoring and establish the ADC channel between the first calling terminal and the network side. In response to the translation start command from the first call terminal, a media stream copying request is initiated to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element. The media processing function network element translates the uplink audio stream to obtain a subtitle stream, and then returns the subtitle stream to the media function network element. The subtitle stream is sent to the first call terminal through the media function network element, so that the first call terminal combines the subtitle stream with the downlink video stream to obtain a subtitle video stream and displays it; or, the subtitle stream is combined with the downlink video stream through the media function network element to obtain a subtitle video stream and sends the subtitle video stream to the first call terminal for display. The triggering network side completes audio and video media anchoring and establishes an ADC channel between the first calling terminal and the network side, which specifically includes: The intelligent translation mini-program triggers the 5G enhanced call SDK to request the establishment of the ADC channel from the core network side, so that the core network side sends a media change call event notification to the intelligent translation application server through the data channel signaling function network element. The intelligent translation application server calls the call control northbound interface provided by the data channel signaling function network element to trigger the network side to perform audio and video media anchor renegotiation and complete the establishment of the ADC channel. The ADC channel includes a first ADC channel for service control and a second ADC channel for subtitle reception. The step of responding to the translation start command from the first call terminal by initiating a media stream copying request to the data channel signaling function network element through the intelligent translation application server, thereby enabling the media function network element to transmit the uplink audio stream from the second call terminal to the media processing function network element, specifically includes: The intelligent translation mini-program triggers the 5G enhanced call SDK to call the first ADC channel to send the start translation command to the media function network element, so that the media function network element reports the start translation command to the intelligent translation application server; The intelligent translation application server initiates a media stream copying request to the data channel signaling function network element, enabling the data channel signaling function network element and the media function network element to negotiate and complete the media stream copying, and then the media function network element transmits the copied uplink audio stream to the media processing function network element. The data channel signaling function network element returns the media stream control address to the intelligent translation application server.

2. The method for displaying translated subtitles during a phone call according to claim 1, characterized in that, The method for displaying translated subtitles during a call also includes the following steps: The intelligent translation mini-program triggers the 5G enhanced call SDK to request the media function network element to obtain the user subscription relationship of the first call terminal, so that the media function network element obtains the user subscription relationship from the intelligent translation application server and returns the user subscription relationship to the intelligent translation mini-program; The intelligent translation mini-program determines whether the first calling terminal has subscribed to the subtitle translation service based on the user's subscription relationship. If the first calling terminal has not subscribed to the subtitle translation service, the intelligent translation mini-program guides the user to activate the service.

3. The method for displaying translated subtitles during a phone call according to claim 1, characterized in that, The process of translating the uplink audio stream into a subtitle stream through the media processing function network element and returning the subtitle stream to the media function network element specifically includes: The intelligent translation application server initiates an intelligent translation control request to the media processing function network element based on the media stream control address, and the intelligent translation control request carries the subtitle receiving address. The media processing function network element performs speech recognition and translation operations on the uplink audio stream to obtain the subtitle stream, and returns the subtitle stream to the media function network element according to the subtitle receiving address.

4. A method for displaying translated subtitles during phone calls according to any one of claims 1 to 3, characterized in that, The process of combining the subtitle stream and the downlink video stream specifically includes: The subtitle stream is parsed to obtain multiple subtitle texts and their corresponding first timestamps; The downlink video stream is decoded to obtain multiple video frames and corresponding second timestamps; The subtitle text and the video frame are matched based on the first timestamp and the second timestamp to obtain the matched target video frame and target subtitle text; A subtitle image is generated based on a preset subtitle style template and the target subtitle text, and the subtitle image is superimposed on the target video frame to obtain a subtitle video frame; The subtitle video frames are video encoded to obtain the subtitle video stream.

5. The method for displaying translated subtitles in a phone call according to claim 4, characterized in that, The step of matching the subtitle text and the video frame based on the first timestamp and the second timestamp to obtain the matched target video frame and target subtitle text specifically includes: Multiple candidate video frames corresponding to the target subtitle text are determined based on the first timestamp and the second timestamp; The candidate video frames are subjected to speech recognition to obtain text content, and the cosine similarity between the text content and the target subtitle text is calculated. When the cosine similarity is greater than or equal to a preset first threshold, the candidate video frame is determined to be the target video frame.

6. A call translation subtitle display system, characterized in that, A method for displaying translated subtitles for phone calls as described in any one of claims 1 to 5, comprising: The channel establishment module is used to trigger the network side to complete the audio and video media anchoring and establish the ADC channel between the first call terminal and the network side. The audio stream copying module is used to respond to the translation start command of the first call terminal and initiate a media stream copying request to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element. The audio stream translation module is used to translate the uplink audio stream into a subtitle stream through the media processing function network element, and return the subtitle stream to the media function network element; The subtitle video synthesis module is used to send the subtitle stream to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream and the downlink video stream to obtain a subtitle video stream and displays it; or, it synthesizes the subtitle stream and the downlink video stream through the media function network element to obtain a subtitle video stream and sends the subtitle video stream to the first call terminal for display.

7. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the call translation subtitle display method as described in any one of claims 1 to 5.

8. A storage medium, said storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the call translation subtitle display method as described in any one of claims 1 to 5.

Images