Call translation subtitle display method and system, electronic equipment and storage medium

Abstract

The invention discloses a call translation subtitle display method and system, electronic equipment and a storage medium, and the method comprises the steps: triggering a network side to complete the anchoring of audio and video media, and building an ADC channel between a first call terminal and the network side; in response to a translation starting command of the first call terminal, initiating a media stream copying request to a data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits an uplink audio stream of a second call terminal to a media processing function network element; translating the uplink audio stream through the media processing function network element to obtain a subtitle stream, and returning the subtitle stream to the media processing function network element; and synthesizing the subtitle stream and the downlink video stream to obtain a subtitle video stream, and displaying the subtitle video stream. The communication efficiency and the information acquisition accuracy are improved, and the method can be widely applied to the technical field of communication.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method, system, electronic device and storage medium for displaying call translation subtitles. Background Art

[0002] With the popularization and application of 5G networks, mobile communication technology has become more efficient and convenient based on the development of 5G enhanced real-time communication technology, improving the quality of real-time calls and user experience. The process of globalization is accelerating, and the demand for cross-language communication is increasing, but using different languages ​​in calls may cause communication difficulties. In traditional methods, playing the translated audio after voice recognition translation will change the original call audio, resulting in information errors, affecting call efficiency and the accuracy of information acquisition.

[0003] Terminology explanation:

[0004] BDC (Bootstrap Data Channel): established between the terminal and the IMS network, used by the terminal to obtain a list of data channel applications from the IMS network and download required data channel applications.

[0005] ADC (Application Data Channel): established between the terminal and the IMS network or between the terminals, and used to transmit application data of the data channel application between the two ends. Summary of the invention

[0006] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0007] To this end, an object of an embodiment of the present invention is to provide a method for displaying call translation subtitles, which improves call efficiency and accuracy of information acquisition.

[0008] Another object of an embodiment of the present invention is to provide a call translation subtitle display system.

[0009] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0010] On the one hand, an embodiment of the present invention provides a method for displaying call translation subtitles, comprising the following steps:

[0011] Triggering the network side to complete audio and video media anchoring, and establishing an ADC channel between the first call terminal and the network side;

[0012] In response to the translation start command of the first call terminal, a media stream replication request is initiated to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element;

[0013] The uplink audio stream is translated by the media processing function network element to obtain a subtitle stream, and the subtitle stream is returned to the media function network element;

[0014] The subtitle stream is sent to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream with the downstream video stream to obtain the subtitle video stream and display it. Alternatively, the subtitle stream is synthesized with the downstream video stream through the media function network element to obtain the subtitle video stream, and the subtitle video stream is sent to the first call terminal for display.

[0015] Further, in one embodiment of the present invention, the triggering network side completes the audio and video media anchoring and establishes an ADC channel between the first call terminal and the network side, which specifically includes:

[0016] The 5G enhanced call SDK is triggered through the smart translation applet to request the core network side to establish the ADC channel, so that the core network side sends a media change call event notification to the smart translation application server through the data channel signaling function network element;

[0017] The intelligent translation application server calls the call control northbound interface provided by the data channel signaling function network element to trigger the network side to perform audio and video media anchor re-negotiation, and complete the establishment of the ADC channel through negotiation;

[0018] The ADC channels include a first ADC channel for service control and a second ADC channel for subtitle reception.

[0019] Furthermore, in one embodiment of the present invention, the method for displaying call translation subtitles further includes the following steps:

[0020] The 5G enhanced call SDK is triggered by the smart translation applet to request the media function network element to obtain the user subscription relationship of the first call terminal, so that the media function network element obtains the user subscription relationship from the smart translation application server and returns the user subscription relationship to the smart translation applet;

[0021] The smart translation applet determines whether the first call terminal has subscribed to the subtitle translation service according to the user subscription relationship. When the first call terminal has not subscribed to the subtitle translation service, the smart translation applet guides the user to activate the service.

[0022] Further, in one embodiment of the present invention, in response to the start translation command of the first call terminal, initiating a media stream replication request to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element, which specifically includes:

[0023] The 5G enhanced call SDK is triggered by the smart translation applet to call the first ADC channel to send the start translation command to the media function network element, so that the media function network element reports the start translation command to the smart translation application server;

[0024] Initiate the media stream replication request to the data channel signaling function network element through the intelligent translation application server, so that the data channel signaling function network element negotiates with the media function network element to complete the media stream replication, and transmit the replicated uplink audio stream to the media processing function network element through the media function network element;

[0025] The media stream control address is returned to the intelligent translation application server through the data channel signaling function network element.

[0026] Further, in one embodiment of the present invention, the translating the uplink audio stream by the media processing function network element to obtain a subtitle stream, and returning the subtitle stream to the media function network element specifically includes:

[0027] Initiate an intelligent translation control request to the media processing function network element through the intelligent translation application server according to the media stream control address, wherein the intelligent translation control request carries a subtitle receiving address;

[0028] The uplink audio stream is subjected to speech recognition and translation operations by the media processing function network element to obtain the subtitle stream, and the subtitle stream is returned to the media function network element according to the subtitle receiving address.

[0029] Further, in one embodiment of the present invention, the synthesizing the subtitle stream and the downstream video stream specifically includes:

[0030] Parsing the subtitle stream to obtain a plurality of subtitle texts and corresponding first timestamps;

[0031] Decoding the downlink video stream to obtain a plurality of video frames and corresponding second timestamps;

[0032] Matching the subtitle text and the video frame according to the first timestamp and the second timestamp to obtain a matched target video frame and a target subtitle text;

[0033] Generate a subtitle image according to a preset subtitle style template and the target subtitle text, and superimpose the subtitle image with the target video frame to obtain a subtitle video frame;

[0034] The subtitle video frame is subjected to video encoding to obtain the subtitle video stream.

[0035] Further, in one embodiment of the present invention, matching the subtitle text and the video frame according to the first timestamp and the second timestamp to obtain a matched target video frame and a target subtitle text specifically includes:

[0036] Determine a plurality of candidate video frames corresponding to the target subtitle text according to the first timestamp and the second timestamp;

[0037] Performing speech recognition on the candidate video frame to obtain text content, and calculating the cosine similarity between the text content and the target subtitle text;

[0038] When the cosine similarity is greater than or equal to a preset first threshold, the candidate video frame is determined to be the target video frame.

[0039] On the other hand, an embodiment of the present invention provides a call translation subtitle display system, including:

[0040] A channel establishment module, used to trigger the network side to complete the audio and video media anchoring, and establish an ADC channel between the first call terminal and the network side;

[0041] An audio stream duplication module, configured to respond to the translation start command of the first call terminal, initiate a media stream duplication request to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element;

[0042] An audio stream translation module, configured to translate the uplink audio stream through the media processing function network element to obtain a subtitle stream, and return the subtitle stream to the media function network element;

[0043] A subtitle video synthesis module is used to send the subtitle stream to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream with the downstream video stream to obtain the subtitle video stream and display it, or synthesizes the subtitle stream with the downstream video stream through the media function network element to obtain the subtitle video stream, and sends the subtitle video stream to the first call terminal for display.

[0044] On the other hand, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein the program, when executed by the processor, realizes the call translation subtitle display method as described above.

[0045] On the other hand, an embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the call translation subtitle display method as described above.

[0046] The advantages and beneficial effects of the present invention will be partly given in the following description, partly become apparent from the following description, or be understood through the practice of the present invention:

[0047] The embodiment of the present invention triggers the network side to complete the audio and video media anchoring, and establishes an ADC channel between the first call terminal and the network side. In response to the start translation command of the first call terminal, a media stream replication request is initiated to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element, and the uplink audio stream is translated by the media processing function network element to obtain a subtitle stream, and the subtitle stream is returned to the media function network element, and the subtitle stream is sent to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream with the downlink video stream to obtain the subtitle video stream and display it, or synthesizes the subtitle stream with the downlink video stream through the media function network element to obtain the subtitle video stream, and sends the subtitle video stream to the first call terminal for display. The embodiment of the present invention superimposes translated subtitles in a video call, and realizes translation by displaying the subtitles on the terminal screen, without affecting the original call audio, reducing information errors, and improving call efficiency and the accuracy of information acquisition; based on the operator's existing IMS core network, system enhancement and business function enhancement are performed, and translated subtitles are displayed during the video call of the user terminal, without the need to install additional translation software, thereby simplifying the user operation process and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solution of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1A flowchart of a method for displaying call translation subtitles provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of an application scenario of the method for displaying call translation subtitles provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of instructions for a method for displaying call translation subtitles provided by an embodiment of the present invention;

[0052] Figure 4 A step flow chart of step S101 provided in an embodiment of the present invention;

[0053] Figure 5 A step flow chart of step S102 provided in an embodiment of the present invention;

[0054] Figure 6 A step flow chart of step S103 provided in an embodiment of the present invention;

[0055] Figure 7 A step flow chart of step S104 provided in an embodiment of the present invention;

[0056] Figure 8 A step flow chart of step S1043 provided in an embodiment of the present invention;

[0057] Fig. 9 A structural diagram of a call translation subtitle display system provided by an embodiment of the present invention;

[0058] Fig.10 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention;

[0059] Fig.11 A schematic diagram of the structure of a storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limitations on the present application. It should be noted that, although the functional module division is performed in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order from the module division in the system schematic diagram or the flow chart. For the step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0061] In the description of the present invention, the meaning of "a plurality" is two or more. If there is a description of the first or the second, it is only for the purpose of distinguishing the technical features, and it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those commonly understood by technicians in the technical field of this application. The terms used in this document are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0062] The method for displaying call translation subtitles provided in the embodiment of the present application can be applied to a terminal, or to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, etc.; the server can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for displaying call translation subtitles, etc., but is not limited to the above forms.

[0063] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0065] First, the application scenario of the embodiment of the present invention is described. Figure 2 The following is a schematic diagram of an application scenario of the call translation subtitle display method provided by an embodiment of the present invention, including a core network side, an enhanced call network side, a terminal side, and an operating environment side. The relevant descriptions are as follows:

[0066] 1) Core network side: Operator's existing communication network elements. EPS / 5GS, used to carry data and call control signals. P-CSCF / BAC, responsible for establishing and maintaining call sessions between user equipment and IMS network. I / S-CSCF, used to receive and process SIP call requests and assume call control functions. MMTel AS, used to provide application services for multimedia telephone services (such as VoLTE, Video over LTE, etc.). HSS / UDM, a central repository for user data, supports functions such as authentication, subscription management, mobility management, session establishment, etc. ENUM / DNS, used to map phone numbers to Internet resource identifiers.

[0067] 2) Data Channel Signaling Function (DCSF): An application that supports access management of 5G enhanced real-time communication services. It has multiple functions, including providing atomic operation capabilities of media functions (MF) to business application services (AS), subscribing to call events, integrating subscription events to MMTel AS, and passing call-related information to business AS. In addition, DCSF also supports storing user information in HSS / UDM, selecting MPF addresses, storing DC application clients (DCAR functions), and controlling the processing of BootstrapDC (BDC).

[0068] 3) Media Function (MF): A media processing application with multiple functions. It can receive command control from MMTelAS and DCSF network elements, including the creation, update and deletion of media resources, media stream replication, audio and video basic capabilities, media synthesis, etc. In addition, MF supports the download and caching of media material resources, and supports data channel interaction with DCSF network elements.

[0069] 4) Media Processing Function (MPF): Provides richer media processing capabilities, supports speech recognition to text, translation, and other AI capabilities.

[0070] 5) Intelligent Translation Application Server (Service AS): A key component responsible for implementing business logic and interacting with DCSF network elements. During registration, the Service AS needs to carry a device-level subscription indication to inform DCSF of call events that need to be reported, and supports instructing the media capability platform to perform the required media processing. The Service AS provides the business logic and application interaction content of the DC application server. The Service AS can effectively manage business processes and collaborate with other systems to complete the intelligent translation business logic control.

[0071] 6) Intelligent Translation Mini Program: Provides an H5 front-end operation interface to realize intelligent translation start and stop control and style configuration, etc.

[0072] 7) 5G enhanced call SDK: provides capabilities such as application data channel creation and closing, DC data sending, terminal rendering, etc., and serves as a bridge for data interaction between the smart translation applet and the enhanced call network side.

[0073] In the embodiment of the present invention, after user terminal A and user terminal B establish a call, the translation function is started through the intelligent translation applet. The applet establishes a data channel through the 5G enhanced call SDK (hereinafter referred to as SDK) and obtains the user's subscription information to confirm whether the user has subscribed to the translation service. If not, guide the user to perform the subscription and billing steps; if already subscribed, start the translation service. The intelligent translation application server (hereinafter referred to as service AS) requests media stream replication and initiates a translation request to the media capability platform (hereinafter referred to as MPF), and sends the processing result to the terminal for display. After the translation is completed, the translation result subtitles are synthesized and displayed on the terminal, providing users with a convenient cross-language call experience.

[0074] like Figure 1 FIG. 1 is a flowchart of a method for displaying call translation subtitles provided by an embodiment of the present invention. Figure 3 FIG. 1 is a schematic diagram of instructions for a method for displaying call translation subtitles provided by an embodiment of the present invention, referring to FIG. Figure 1 and 3 The embodiment of the present invention provides a method for displaying call translation subtitles, which specifically includes the following steps:

[0075] S101. Trigger the network side to complete audio and video media anchoring, and establish an ADC channel between the first call terminal and the network side.

[0076] like Figure 4 FIG. 1 is a flowchart of step S101 provided in an embodiment of the present invention, referring to FIG. Figure 4 As an optional implementation, the network side is triggered to complete the audio and video media anchoring and establish an ADC channel between the first call terminal and the network side, which specifically includes:

[0077] S1011. Triggering the 5G enhanced call SDK through the smart translation applet to request the core network side to establish an ADC channel, so that the core network side sends a media change call event notification to the smart translation application server through the data channel signaling function network element;

[0078] S1012, calling the call control northbound interface provided by the data channel signaling function network element through the intelligent translation application server to trigger the network side to perform audio and video media anchor re-negotiation, and complete the establishment of the ADC channel through negotiation;

[0079] The ADC channels include a first ADC channel for service control and a second ADC channel for subtitle reception.

[0080] Specifically, the smart translation applet triggers the establishment of an ADC channel through the 5G enhanced call SDK. After the service AS receives the media change call event notification, it calls the call control northbound interface provided by DCSF to trigger the network side to complete the re-negotiation of audio and video media anchoring, and negotiates to complete the establishment of the ADC between terminal A and the network side. The smart translation service needs to establish two ADCs, ADC1 for service control and ADC2 for subtitle reception.

[0081] As an optional implementation, the call translation subtitle display method further includes the following steps:

[0082] The 5G enhanced call SDK is triggered through the smart translation applet to request the media function network element to obtain the user subscription relationship of the first call terminal, so that the media function network element obtains the user subscription relationship from the smart translation application server and returns the user subscription relationship to the smart translation applet;

[0083] The smart translation applet determines whether the first call terminal has subscribed to the subtitle translation service based on the user's subscription relationship. When the first call terminal has not subscribed to the subtitle translation service, the smart translation applet guides the user to activate the service.

[0084] Specifically, the smart translation applet requests the MF to obtain the current user subscription relationship through the SDK. The MF obtains the user's current subscription relationship from the smart translation AS and sends the obtained user's current subscription relationship information to the SDK through ADC1, and displays it in the smart translation applet. If the user has not subscribed, guide the user to complete the subscription and billing process; if the user has subscribed, start the translation function.

[0085] S102. In response to the translation start command of the first call terminal, a media stream replication request is initiated to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element.

[0086] like Figure 5 FIG. 1 is a flow chart of step S102 provided in an embodiment of the present invention, referring to FIG. Figure 5 As an optional implementation, in response to the translation start command of the first call terminal, a media stream replication request is initiated to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element, which specifically includes:

[0087] S1021. Triggering the 5G enhanced call SDK to call the first ADC channel through the smart translation applet to send a start translation command to the media function network element, so that the media function network element reports the start translation command to the smart translation application server;

[0088] S1022, initiating a media stream replication request to the data channel signaling function network element through the intelligent translation application server, so that the data channel signaling function network element and the media function network element negotiate to complete the media stream replication, and transmit the replicated uplink audio stream to the media processing function network element through the media function network element;

[0089] S1023. Return the media stream control address to the intelligent translation application server through the data channel signaling function network element.

[0090] Specifically, the smart translation applet sends a translation start request by requesting SDK to call ADC1. After receiving the request, SDK sends a translation start command to MF. After receiving the instruction, MF reports it to the smart translation application server. The smart translation AS initiates a media stream replication request to DCSF. MF negotiates with DCSF to complete the media stream replication and copies a copy of the terminal B uplink audio stream from MF to MPF. DCSF returns information such as the media stream control address URL1 to the service AS based on the replication result.

[0091] S103: The uplink audio stream is translated by the media processing function network element to obtain a subtitle stream, and the subtitle stream is returned to the media function network element.

[0092] like Figure 6 FIG. 1 is a flow chart of step S103 provided in an embodiment of the present invention, referring to FIG. Figure 6 As an optional implementation, the uplink audio stream is translated into a subtitle stream by a media processing function network element, and the subtitle stream is returned to the media function network element, which specifically includes:

[0093] S1031, initiating an intelligent translation control request to the media processing function network element through the intelligent translation application server according to the media stream control address, where the intelligent translation control request carries a subtitle receiving address;

[0094] S1032. Perform speech recognition and translation operations on the uplink audio stream through the media processing function network element to obtain a subtitle stream, and return the subtitle stream to the media function network element according to the subtitle receiving address.

[0095] Specifically, the service AS initiates an intelligent translation control request to URL1 (MPF), instructing MPF to perform speech recognition and translation operations. The request carries the subtitle receiving address URL2, which is used by MPF to respond to the translation results to MF later. The service AS starts recording billing calls. The service AS sends a successful translation start request through ADC1, which is transparently transmitted through intermediate network elements such as MF and SDK, and reported to the intelligent translation applet. The applet completes the prompt of the user's operation results. MPF performs language recognition and translation on the upstream audio stream of terminal B, and returns the translation result to MF according to the subtitle receiving address URL2.

[0096] S104. Send the subtitle stream to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream with the downstream video stream to obtain the subtitle video stream and display it. Alternatively, synthesize the subtitle stream with the downstream video stream through the media function network element to obtain the subtitle video stream, and send the subtitle video stream to the first call terminal for display.

[0097] Specifically, the present invention provides the following two methods for subtitle rendering:

[0098] The SDK on the terminal side renders, and MF uses ADC2 to send the translation result subtitle stream to the SDK of terminal A. The SDK receives and synthesizes the subtitle stream for display on terminal A.

[0099] Network-side rendering: MF combines the subtitle stream content and subtitle template to complete the synthesis of the downstream video stream of terminal A, and sends the video stream with synthesized subtitles to terminal A for display.

[0100] By using the cosine similarity algorithm (subtitle content matching algorithm) to match subtitle content and applying the Alpha synthesis algorithm (subtitle style synthesis algorithm) to synthesize subtitle styles, the synthesized subtitles are superimposed on the original video stream through the superposition fusion algorithm (superposition algorithm), and the H.264 video encoding algorithm is used to encode and compress the processed video to ensure efficient transmission of subtitled video streams. This technology includes subtitle content matching algorithm, subtitle style synthesis algorithm and superposition algorithm, which realizes efficient synthesis of subtitled videos and improves the user's audio-visual experience.

[0101] like Figure 7 FIG. 1 is a flowchart of step S104 provided in an embodiment of the present invention, referring to FIG. Figure 7 As an optional implementation, the subtitle stream and the downstream video stream are synthesized, which specifically includes:

[0102] S1041, parsing the subtitle stream to obtain a plurality of subtitle texts and corresponding first timestamps;

[0103] S1042. Decode the downlink video stream to obtain multiple video frames and corresponding second timestamps;

[0104] S1043, matching the subtitle text and the video frame according to the first timestamp and the second timestamp to obtain a matched target video frame and a target subtitle text;

[0105] S1044, generating a subtitle image according to a preset subtitle style template and a target subtitle text, and superimposing the subtitle image with a target video frame to obtain a subtitle video frame;

[0106] S1045: Perform video encoding on the subtitle video frame to obtain a subtitle video stream.

[0107] Specifically, the FFmpeg tool is first used to preprocess the subtitle stream and video stream, which includes parsing the subtitles to extract the text and the corresponding timestamps, and decoding the video to decompose it into a series of frames with timestamps. Then, subtitle content matching is performed, the goal of which is to match the text in the subtitle stream to the corresponding frame in the video stream, which is achieved by comparing the timestamps in the subtitle stream and the video stream. Next, subtitle style synthesis is performed according to the style template. The goal of this step is to generate a subtitle image. The style template defines the location, font, color, size, background and other attributes of the subtitle. First, an image containing the subtitle text is generated according to the style template, and then the Alpha synthesis algorithm is used to synthesize this image and a transparent background to generate the final subtitle image. Then, the synthesized subtitle image is superimposed on the original video stream. This step is achieved by using the image fusion function cv2.addWeighted of OpenCV-Python, inputting the subtitle image, video frame, image weight and output scalar to complete the image synthesis. Finally, the video frame with subtitles added is re-encoded into a video stream and output. This step is completed by using the H.264 video encoding algorithm and the ffmpeg.output function of the FFmpeg tool.

[0108] like Figure 8 FIG. 1 is a flow chart of step S1043 provided in an embodiment of the present invention, referring to FIG. Figure 8 As an optional implementation, the subtitle text and the video frame are matched according to the first timestamp and the second timestamp to obtain a matched target video frame and a target subtitle text, which specifically includes:

[0109] S10431, determining a plurality of candidate video frames corresponding to the target subtitle text according to the first timestamp and the second timestamp;

[0110] S10432, performing speech recognition on the candidate video frame to obtain text content, and calculating the cosine similarity between the text content and the target subtitle text;

[0111] S10433: When the cosine similarity is greater than or equal to a preset first threshold, determine that the candidate video frame is a target video frame.

[0112] Specifically, in order to further optimize the subtitle content matching, the cosine similarity algorithm can also be used. If the cosine similarity between the text contained in a certain frame in the video stream and the subtitle text in the subtitle stream is higher than a certain threshold, it is considered that the frame corresponds to this subtitle.

[0113] In some optional embodiments, after user terminal A establishes a call with user terminal B, the smart translation applet establishes a data channel through the SDK, obtains the user's subscription information and verifies the translation service subscription status. If not subscribed, guide the user to complete the subscription and billing process; if subscribed, start the translation function. The service AS sends a media stream copy request to the MPF and initiates a translation request, and the MPF returns the translated subtitle stream to the MF. The MF determines whether the target terminal supports the terminal-side rendering function based on the call signaling parameters. For terminals that support rendering, the SDK completes the subtitle display; for terminals that do not support rendering, the MF completes the subtitle synthesis of the downlink video and sends the video stream with subtitles. This method effectively utilizes terminal resources and reduces the subtitle rendering burden of the MF network element on the network side, but requires communication terminal manufacturers to adapt to support terminal-side rendering.

[0114] In some optional embodiments, after the call between user terminal A and user terminal B is connected, the smart translation applet establishes a data channel through the SDK to obtain the user's subscription information and verify the translation service subscription status. If it has not been subscribed, guide the user to complete the subscription and billing process; if it has been subscribed, start the translation function. The service AS sends a media stream copy request to the MPF and initiates a translation request. The MPF returns the translated subtitle stream to the MF. The MF completes the subtitle synthesis of the downlink video and sends the subtitled video stream to the current terminal. This solution only has basic network-side subtitle rendering capabilities, and the difficulty and cost of implementation are relatively low.

[0115] The above describes the method flow and implementation process of the embodiment of the present invention. It can be understood that in order to ensure the stability of the basic media synthesis capability of MF and provide a more flexible access path for AI functions that may change frequently, the embodiment of the present invention will separate the AI ​​functions that are more likely to change, such as speech recognition to text and translation, from MF as an independent media processing function (MPF) service. Such a design can achieve more convenient access and docking of third-party AI capabilities, while ensuring the stable operation of the basic media synthesis capabilities of MF. The MPF service will independently undertake functions such as speech recognition to text and translation, and provide customers and partners with standard interfaces and open access specifications to ensure efficient collaboration and integration. MF still focuses on providing basic media processing capabilities, such as audio and video resource management, media stream replication and basic capability support, to maintain its reliability and stability. Through this architectural design, the embodiment of the present invention can better cope with the iteration of AI function changes and technological innovations, while providing a more flexible and scalable solution for the business.

[0116] The embodiment of the present invention superimposes translated subtitles in a video call, and realizes translation by displaying the subtitles on the terminal screen, without affecting the original call audio, reducing information errors, and improving call efficiency and the accuracy of information acquisition; based on the operator's existing IMS core network, system enhancement and business function enhancement are performed, and translated subtitles are displayed during the video call of the user terminal, without the need to install additional translation software, thereby simplifying the user operation process and improving the user experience.

[0117] like Fig. 9 FIG. 1 is a schematic diagram of the structure of a call translation subtitle display system provided by an embodiment of the present invention, referring to FIG. Fig. 9 , an embodiment of the present invention provides a call translation subtitle display system, comprising:

[0118] A channel establishment module, used to trigger the network side to complete the audio and video media anchoring, and establish an ADC channel between the first call terminal and the network side;

[0119] An audio stream duplication module, configured to respond to a translation start command of the first call terminal, initiate a media stream duplication request to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element;

[0120] The audio stream translation module is used to translate the uplink audio stream through the media processing function network element to obtain a subtitle stream, and return the subtitle stream to the media function network element;

[0121] The subtitle video synthesis module is used to send the subtitle stream to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream with the downstream video stream to obtain the subtitle video stream and display it, or synthesizes the subtitle stream with the downstream video stream through the media function network element to obtain the subtitle video stream, and sends the subtitle video stream to the first call terminal for display.

[0122] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0123] The embodiment of the present invention also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned call translation subtitle display method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0124] like Fig.10 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention, referring to FIG. Fig.10 , an embodiment of the present invention provides an electronic device, including:

[0125] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0126] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the call translation subtitle display method of the embodiment of the present invention;

[0127] Input / output interface 1003, used to implement information input and output;

[0128] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0129] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0130] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0131] like Fig.11 FIG. 1 is a schematic diagram of a storage medium according to an embodiment of the present invention. Fig.11 The embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs 1101, and the one or more programs 1101 can be executed by one or more processors to implement the above-mentioned call translation subtitle display method.

[0132] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The embodiment of the present invention also discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown.

[0134] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0135] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified to the contrary, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0136] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0137] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0138] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the above-mentioned program is printed, since the above-mentioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in other suitable ways as necessary, and then stored in a computer memory.

[0139] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0140] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0141] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0142] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for displaying call translation subtitles, characterized in that: The following steps are involved: Triggering the network side to complete audio and video media anchoring, and establishing an ADC channel between the first call terminal and the network side; In response to the translation start command of the first call terminal, a media stream replication request is initiated to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element; The uplink audio stream is translated by the media processing function network element to obtain a subtitle stream, and the subtitle stream is returned to the media function network element; The subtitle stream is sent to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream with the downstream video stream to obtain the subtitle video stream and display it. Alternatively, the subtitle stream is synthesized with the downstream video stream through the media function network element to obtain the subtitle video stream, and the subtitle video stream is sent to the first call terminal for display.

2. A method for displaying call translation subtitles according to claim 1, characterized in that: The triggering network side to complete the audio and video media anchoring and establish the ADC channel between the first call terminal and the network side specifically includes: The 5G enhanced call SDK is triggered through the smart translation applet to request the core network side to establish the ADC channel, so that the core network side sends a media change call event notification to the smart translation application server through the data channel signaling function network element; The intelligent translation application server calls the call control northbound interface provided by the data channel signaling function network element to trigger the network side to perform audio and video media anchor re-negotiation, and complete the establishment of the ADC channel through negotiation; The ADC channels include a first ADC channel for service control and a second ADC channel for subtitle reception.

3. A method for displaying call translation subtitles according to claim 2, characterized in that: The call translation subtitle display method also includes the following steps: The 5G enhanced call SDK is triggered by the smart translation applet to request the media function network element to obtain the user subscription relationship of the first call terminal, so that the media function network element obtains the user subscription relationship from the smart translation application server and returns the user subscription relationship to the smart translation applet; The smart translation applet determines whether the first call terminal has subscribed to the subtitle translation service according to the user subscription relationship. When the first call terminal has not subscribed to the subtitle translation service, the smart translation applet guides the user to activate the service.

4. A method for displaying call translation subtitles according to claim 2, characterized in that: The method of initiating a media stream duplication request to a data channel signaling function network element through an intelligent translation application server in response to a start translation command of the first call terminal, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element, specifically includes: The 5G enhanced call SDK is triggered by the smart translation applet to call the first ADC channel to send the start translation command to the media function network element, so that the media function network element reports the start translation command to the smart translation application server; Initiate the media stream replication request to the data channel signaling function network element through the intelligent translation application server, so that the data channel signaling function network element negotiates with the media function network element to complete the media stream replication, and transmit the replicated uplink audio stream to the media processing function network element through the media function network element; The media stream control address is returned to the intelligent translation application server through the data channel signaling function network element.

5. A method for displaying call translation subtitles according to claim 4, characterized in that: The translating the uplink audio stream by the media processing function network element to obtain a subtitle stream, and returning the subtitle stream to the media function network element specifically includes: Initiate an intelligent translation control request to the media processing function network element through the intelligent translation application server according to the media stream control address, wherein the intelligent translation control request carries a subtitle receiving address; The uplink audio stream is subjected to speech recognition and translation operations by the media processing function network element to obtain the subtitle stream, and the subtitle stream is returned to the media function network element according to the subtitle receiving address.

6. A method for displaying call translation subtitles according to any one of claims 1 to 5, characterized in that: The synthesizing the subtitle stream and the downlink video stream specifically includes: Parsing the subtitle stream to obtain a plurality of subtitle texts and corresponding first timestamps; Decoding the downlink video stream to obtain a plurality of video frames and corresponding second timestamps; Matching the subtitle text and the video frame according to the first timestamp and the second timestamp to obtain a matched target video frame and a target subtitle text; Generate a subtitle image according to a preset subtitle style template and the target subtitle text, and superimpose the subtitle image with the target video frame to obtain a subtitle video frame; The subtitle video frame is subjected to video encoding to obtain the subtitle video stream.

7. A method for displaying call translation subtitles according to claim 6, characterized in that: The step of matching the subtitle text and the video frame according to the first timestamp and the second timestamp to obtain a matched target video frame and a matched target subtitle text specifically includes: Determine a plurality of candidate video frames corresponding to the target subtitle text according to the first timestamp and the second timestamp; Performing speech recognition on the candidate video frame to obtain text content, and calculating the cosine similarity between the text content and the target subtitle text; When the cosine similarity is greater than or equal to a preset first threshold, the candidate video frame is determined to be the target video frame.

8. A call translation subtitle display system, characterized in that: include: A channel establishment module, used to trigger the network side to complete the audio and video media anchoring, and establish an ADC channel between the first call terminal and the network side; An audio stream replication module, configured to respond to the translation start command of the first call terminal, initiate a media stream replication request to the data channel signaling function network element through the intelligent translation application server, so that the media function network element transmits the uplink audio stream of the second call terminal to the media processing function network element; An audio stream translation module, configured to translate the uplink audio stream through the media processing function network element to obtain a subtitle stream, and return the subtitle stream to the media function network element; A subtitle video synthesis module is used to send the subtitle stream to the first call terminal through the media function network element, so that the first call terminal synthesizes the subtitle stream with the downstream video stream to obtain the subtitle video stream and display it, or synthesizes the subtitle stream with the downstream video stream through the media function network element to obtain the subtitle video stream, and sends the subtitle video stream to the first call terminal for display.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the call translation subtitle display method as described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the call translation subtitle display method according to any one of claims 1 to 7.

Images