Method and device for displaying information in call, equipment, medium and product

Abstract

The invention relates to the technical field of data processing, and particularly provides a method, a device, equipment, a medium and a product for displaying information in a call, and the method comprises the steps: if to-be-displayed information of a first user terminal is detected when the first user terminal is in a video call state, extracting text description information of the to-be-displayed information; wherein the to-be-displayed information is the to-be-displayed information sent by a second user terminal to the first user terminal; determining a call video stream to be displayed of the first user terminal at the next moment; and performing video subtitle synthesis based on the text description information and the call video stream to obtain a target video stream, and issuing the target video stream to the first user terminal for display.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, equipment, medium and product for displaying information during a call. Background Art

[0002] With the continuous development of communication technology, more and more users are making calls through 5G. Compared with traditional call services, 5G calls provide more video call services.

[0003] When a user is making a video call, the images captured by the other party's camera and the audio captured by the microphone will be displayed on both screens in real time. That is, the caller displays the video of the called party, and the called party displays the video of the caller. The user terminal generally displays the content of the video call in full screen, and the user generally turns on the external speaker when making a video call and makes the call facing the phone. At this time, when the user terminal that is making a video call receives a message, most user terminals will only display the receipt prompt of the message or will not prompt the message. Therefore, the user to whom the user terminal belongs may miss important messages when making a video call. Summary of the invention

[0004] In view of the above problems, the present disclosure is proposed. The present disclosure provides a method, device, equipment, medium and product for displaying information during a call.

[0005] According to one aspect of the present disclosure, a method for displaying a message during a call is provided, which is applied to a service server and includes:

[0006] If the first user terminal is in a video call state and the information to be displayed of the first user terminal is detected, the text description information of the information to be displayed is extracted; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal;

[0007] Determining a call video stream to be displayed on the first user terminal at a next moment;

[0008] Video subtitles are synthesized based on the text description information and the call video stream to obtain a target video stream, and the target video stream is sent to the first user terminal for display.

[0009] In addition, according to another embodiment of one aspect of the present disclosure, the step of synthesizing video subtitles based on the text description information and the call video stream to obtain a target video stream includes:

[0010] Determining whether the first user terminal meets the video subtitle synthesis condition;

[0011] When the first user terminal meets the video subtitle synthesis condition, the text description information and the call video stream are synthesized into video subtitles to obtain a target video stream.

[0012] In addition, according to another embodiment of one aspect of the present disclosure, the step of synthesizing video subtitles based on the text description information and the call video stream to obtain a target video stream includes:

[0013] Determine the text attributes of the text description information; wherein the text attributes include but are not limited to: the fusion special effect attribute of the text description information and the real-time attribute of the text description information;

[0014] Based on the text attributes, the text description information, the call video stream and the special effect fusion data of the text description information are synthesized into video subtitles to obtain the target video stream.

[0015] In addition, according to another embodiment of one aspect of the present disclosure, based on the text attribute, the text description information, the call video stream, and the special effect fusion data of the text description information are synthesized into video subtitles to obtain the target video stream, including:

[0016] A video subtitle fusion application is sent to the capability network element, so that the capability network element synthesizes the video subtitles with the text description information, the call video stream and the special effect fusion data of the text description information to obtain the target video stream when the text attribute meets the attribute requirements.

[0017] In addition, according to another embodiment of one aspect of the present disclosure, if the first user terminal is in a video call state, the information to be displayed of the first user terminal is detected, and the text description information of the information to be displayed is extracted, including:

[0018] If the first user terminal is in a video call state and the to-be-displayed information of the first user terminal sent by the message center is detected, the text description information of the to-be-displayed information is extracted.

[0019] According to another aspect of the present disclosure, a method for displaying a message during a call is provided and applied to a message center, comprising:

[0020] In the case of detecting information to be displayed of the first user terminal, determining the call state of the first user terminal; wherein the information to be displayed is information to be displayed sent by the second user terminal to the first user terminal;

[0021] In a case where it is determined based on the call status that the first user terminal is in a video call state, the information to be displayed is sent to a service server.

[0022] In addition, according to another embodiment of one aspect of the present disclosure, the determining the call status of the first user terminal includes:

[0023] The call status of the first user terminal is determined based on the status record information of the first user terminal and based on the information content of the status record information.

[0024] In addition, in another embodiment according to one aspect of the present disclosure, the method further includes:

[0025] Acquiring a video call establishment notification sent by the capability network element after detecting a video communication connection between the first user terminal and the third user terminal;

[0026] The information content is modified to a video call status based on the video call establishment notification.

[0027] In addition, in another embodiment according to one aspect of the present disclosure, the method further includes:

[0028] Acquiring a video call interruption notification sent by the capability network element after detecting that the first user terminal interrupts the video call connection;

[0029] The information content is modified from the video call state to a call idle state based on the video call interruption notification.

[0030] According to another aspect of the present disclosure, there is provided a device for displaying information during a call, comprising:

[0031] an extraction module, configured to extract text description information of the information to be displayed if the information to be displayed of the first user terminal is detected when the first user terminal is in a video call state; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal;

[0032] A first determining module, used to determine a call video stream to be displayed on the first user terminal at a next moment;

[0033] A synthesis module is used to synthesize video subtitles based on the text description information and the call video stream to obtain a target video stream, and send the target video stream to the first user terminal for display.

[0034] According to another aspect of the present disclosure, there is provided a device for displaying information during a call, comprising:

[0035] A second determination module is used to determine the call status of the first user terminal when the information to be displayed of the first user terminal is detected; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal;

[0036] The sending module is used to send the information to be displayed to the service server when it is determined based on the call status that the first user terminal is in a video call state, so that the service server processes the information to be displayed.

[0037] According to another aspect of the present disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement steps of a method for determining a log template.

[0038] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of a method for determining a log template are implemented.

[0039] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the steps of a method for determining a log template when the computer program / instruction is executed by a processor.

[0040] As described in detail below, according to an embodiment of the present disclosure, a method, device, equipment, medium and product for displaying information during a call. First, if the first user terminal is in a video call state, the information to be displayed of the first user terminal is detected, and the text description information of the information to be displayed is extracted; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal; secondly, the call video stream to be displayed of the first user terminal at the next moment is determined; finally, video subtitles are synthesized based on the text description information and the call video stream to obtain the target video stream, and the target video stream is sent to the first user terminal for display.

[0041] When the user terminal is in a video call state, the text description information of the information to be displayed received in the video call can be extracted. The text description information is synthesized with the call video stream that needs to be displayed on the user terminal at the next moment, so that the information to be displayed is displayed to the user of the user terminal in the form of subtitles. While avoiding the user from missing important information, it will not affect the user's current video call connection, thereby improving the user's experience of the video call.

[0042] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0044] Figure 1 A flowchart of a method for displaying information during a call provided by an embodiment of the present disclosure.

[0045] Figure 2 A flowchart of another method for displaying information during a call provided by an embodiment of the present disclosure.

[0046] Figure 3 An interactive diagram of a call establishment process in a method for displaying in-call information provided by an embodiment of the present disclosure.

[0047] Figure 4 An interactive diagram of video subtitle synthesis processing in a method for displaying information during a call provided by an embodiment of the present disclosure.

[0048] Figure 5 An interactive diagram of an on-hook process in a method for displaying information during a call provided by an embodiment of the present disclosure.

[0049] Figure 6 An architectural diagram of a system for displaying information during a call provided in an embodiment of the present disclosure.

[0050] Figure 7 A schematic diagram of a device for displaying information during a call provided by an embodiment of the present disclosure.

[0051] Figure 8 A schematic diagram of another device for displaying information during a call provided by an embodiment of the present disclosure.

[0052] Fig. 9 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0054] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0055] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0056] Research has found that with the continuous development of communication technology, more and more users are making calls through 5G. Compared with traditional call services, 5G calls provide more video call services.

[0057] When a user is making a video call, the images captured by the other party's camera and the audio captured by the microphone will be displayed on both screens in real time. That is, the caller displays the video of the called party, and the called party displays the video of the caller. The user terminal generally displays the content of the video call in full screen, and the user generally turns on the external speaker when making a video call and makes the call facing the phone. At this time, when the user terminal that is making a video call receives a message, most user terminals will only display the receipt prompt of the message or will not prompt the message. Therefore, the user to whom the user terminal belongs may miss important messages when making a video call.

[0058] Based on the above research, the present disclosure provides a method for displaying information during a call. When a user terminal is in a video call state, text description information of the information to be displayed received during the video call can be extracted. The text description information is synthesized with the call video stream that needs to be displayed on the user terminal at the next moment, so that the information to be displayed is displayed to the user of the user terminal in the form of subtitles. While avoiding the user from missing important information, it will not affect the user's current video call connection, thereby improving the user's experience of the video call.

[0059] To facilitate understanding of this embodiment, a method for displaying information during a call disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the method for displaying information during a call provided in an embodiment of the present disclosure is generally an electronic device with certain computing capabilities. In some possible implementations, the method for displaying information during a call can be implemented by a processor calling computer-readable instructions stored in a memory.

[0060] See also Figure 1 FIG. 1 is a flowchart of a method for displaying information during a call provided by an embodiment of the present disclosure, the method comprising steps S101 to S103, applied to a service server, wherein:

[0061] S101. If the first user terminal is in a video call state and information to be displayed of the first user terminal is detected, text description information of the information to be displayed is extracted; wherein the information to be displayed is information to be displayed sent by the second user terminal to the first user terminal.

[0062] In an embodiment of the present disclosure, a message center (ie, a 5G message center, 5G massage center, 5GMC) may record the call status of the first user terminal, wherein the call status includes: a video call status and a call idle status.

[0063] Here, the message center can determine whether the first user terminal is in a video call state according to the record. When detecting that the first user terminal receives the information to be displayed and determining that the first user terminal is in a video call state, the message center can send the information to be displayed to the service server.

[0064] After receiving the information to be displayed, the service server (ie, service AS, service Application Server) can extract the text description information with the display information. The information to be displayed can be video information or text information.

[0065] Here, in the case where the information to be displayed is video information, the service server may extract text information in the video information as text description information, or summarize the content of the video information and use the summarized text content as text description information.

[0066] In the case that the information to be displayed is text information, the service server may use the text information as the text description information of the information to be displayed, or summarize the text and use the summarized text information as the text description information.

[0067] S102: Determine a call video stream to be displayed on the first user terminal at the next moment.

[0068] In an embodiment of the present disclosure, a call video stream sent by a third user terminal and required to be displayed on a first user terminal at the next moment is determined, wherein the third user terminal is a user terminal currently connected to a video call with the first user terminal.

[0069] S103: Perform video subtitle synthesis based on the text description information and the call video stream to obtain a target video stream, and send the target video stream to the first user terminal for display.

[0070] In an embodiment of the present disclosure, after determining the text description information, the service server may send a video subtitle synthesis processing request to the media plane (ie, VoNR+media plane) through a capability network element (ie, VoNR+capability network element).

[0071] Here, the video subtitle synthesis processing request carries text description information and the call video stream. After the media plane receives the video subtitle synthesis processing request, it can synthesize the video subtitles based on the text description information and the call video stream to obtain the target video stream.

[0072] Here, the service server may call the interface provided by the capability network element to instruct the media plane to perform video subtitle synthesis based on the text description information and the call video stream.

[0073] In an embodiment of the present disclosure, firstly, if the first user terminal is in a video call state and information to be displayed of the first user terminal is detected, text description information of the information to be displayed is extracted; wherein the information to be displayed is information to be displayed sent by the second user terminal to the first user terminal; secondly, the call video stream to be displayed of the first user terminal at the next moment is determined; finally, video subtitles are synthesized based on the text description information and the call video stream to obtain the target video stream, and the target video stream is sent to the first user terminal for display.

[0074] In the above implementation, when the user terminal is in a video call state, the text description information of the information to be displayed received in the video call can be extracted. The text description information is synthesized with the call video stream that needs to be displayed on the user terminal at the next moment, so that the information to be displayed is displayed to the user of the user terminal in the form of subtitles. In the case of avoiding the user missing important information, it will not affect the user's current video call connection, thereby improving the user's experience of the video call.

[0075] In an optional implementation, the above steps perform video subtitle synthesis based on the text description information and the call video stream to obtain the target video stream, specifically including the following steps:

[0076] First, determining whether the first user terminal meets the video subtitle synthesis condition;

[0077] Secondly, when the first user terminal meets the video subtitle synthesis conditions, the text description information and the call video stream are synthesized into video subtitles to obtain the target video stream.

[0078] In an embodiment of the present disclosure, before requesting the media plane to perform video subtitle synthesis based on the text description information and the call video stream information, the service server may determine whether the first user terminal meets the video subtitle synthesis condition.

[0079] Here, the video subtitle synthesis condition includes at least one of the following: the first user terminal subscribes to the video subtitle synthesis service; the first user terminal and the second user terminal are in a video call state; a confirmation synthesis request sent by the first user terminal is received; the importance of the information to be displayed is greater than or equal to the preset importance.

[0080] For example, the service server may determine the video subtitle synthesis condition for the first user terminal when detecting that the first user terminal has subscribed to the video subtitle synthesis service.

[0081] The service server may determine the video subtitle synthesis condition of the first user terminal when detecting that the first user terminal has subscribed to the video subtitle synthesis service and is in a video call state.

[0082] The service server may send a confirmation synthesis prompt to the first user terminal when detecting that the first user terminal has subscribed to the video subtitle synthesis service. Afterwards, the first user terminal may send a confirmation synthesis request to the service server when determining that the video subtitle synthesis service needs to be used. Afterwards, the service server determines the video subtitle synthesis condition of the first user terminal when receiving the confirmation synthesis request.

[0083] The service server may determine the importance of the information to be displayed based on the text description information of the information to be displayed when detecting that the first user terminal has subscribed to the video subtitle synthesis service. Afterwards, the service server determines the video subtitle synthesis condition of the first user terminal when determining that the importance of the information to be displayed is greater than or equal to the preset importance.

[0084] In an optional implementation, the above steps perform video subtitle synthesis based on the text description information and the call video stream to obtain the target video stream, specifically including the following steps:

[0085] First, determine the text attributes of the text description information; wherein the text attributes include but are not limited to: fusion special effect attributes of the text description information, and real-time attributes of the text description information;

[0086] Then, based on the text attributes, the text description information, the call video stream and the special effect fusion data of the text description information are synthesized into video subtitles to obtain the target video stream.

[0087] In an embodiment of the present disclosure, when the service server determines that the first user terminal has subscribed to the special effect synthesis service, the text attribute of the text description information is determined.

[0088] Here, the service server may send the text description information to the external media capability platform, and then the external media capability platform may determine the text attribute of the text description information based on the text description information.

[0089] After determining the text attributes of the text description information, the external media capability platform can determine the mapping of special effect fusion data and superposition models corresponding to the text attributes.

[0090] Afterwards, the external media capability platform may send a model download notification to the service server. When the service server receives the model download notification, it may control the media plane to download the superimposed model.

[0091] Here, the service server can control the media plane to download the above-mentioned superposition model through the capability network element. Afterwards, the media plane can synthesize the video subtitles of the text description information, the call video stream and the special effect fusion data of the text description information through the superposition model to obtain the target video stream.

[0092] In an optional implementation, the above steps are based on text attributes, and the special effect fusion data of the text description information, the call video stream and the text description information are synthesized into video subtitles to obtain the target video stream, which specifically includes the following steps:

[0093] A video subtitle fusion application is sent to the capability network element so that the capability network element can synthesize the video subtitles by combining the text description information, the call video stream and the special effect fusion data of the text description information to obtain the target video stream when the text attributes meet the attribute requirements.

[0094] In an embodiment of the present disclosure, the attribute requirement includes at least one of the following: the first user terminal subscribes to a special effect fusion service; the real-time performance of the information to be displayed is less than a preset real-time performance requirement.

[0095] Here, the video subtitle fusion application carries text description information, call video stream and special effect fusion data of the text description information. The capability network element can determine the real-time nature of the information to be displayed based on the text description information.

[0096] Afterwards, when the first user terminal subscribes to the special effects fusion service and the real-time performance of the information to be displayed is less than the preset real-time performance requirement, the special effects fusion data of the text description information, the call video stream and the text description information are synthesized into video subtitles to obtain the target video stream.

[0097] In the above implementation, it takes longer to synthesize the special effect fusion data, text description information and call video stream into video subtitles than to synthesize the text description information and call video stream into video subtitles. When the real-time requirement of the information to be displayed is greater than or equal to the preset real-time requirement, it means that the real-time requirement of the information to be displayed is strong and needs to be displayed in real time on the display interface of the first user terminal. Therefore, when the attribute requirements are not met, only the text description information and the call video stream are synthesized into video subtitles, so that the information to be displayed is displayed faster on the display interface of the first user terminal.

[0098] In an optional implementation manner, if the first user terminal is in a video call state, the above step detects the information to be displayed of the first user terminal, then extracts the text description information of the information to be displayed, specifically including the following steps:

[0099] If the first user terminal is in a video call state and the to-be-displayed information of the first user terminal sent by the message center is detected, text description information of the to-be-displayed information is extracted.

[0100] In an embodiment of the present disclosure, the message center may record the call status of the first user terminal. The message center determines the call status of the first user terminal when determining that the first user terminal receives information to be displayed sent by another user terminal (ie, the second user terminal).

[0101] Here, the message center may query the call status of the first user terminal in the record of the first user terminal. When determining that the call status of the first user terminal is a video call status, the message center may send the information to be displayed to the service server.

[0102] See also Figure 2 FIG. 2 is a flowchart of another method for displaying information during a call provided by an embodiment of the present disclosure, the method comprising steps S201 to S202, and is applied to a message center, wherein:

[0103] S201. When information to be displayed of a first user terminal is detected, determine a call status of the first user terminal; wherein the information to be displayed is information to be displayed sent by a second user terminal to the first user terminal.

[0104] In an embodiment of the present disclosure, the message center may record the call status of the first user terminal. The message center determines the call status of the first user terminal when determining that the first user terminal receives information to be displayed sent by another user terminal (ie, the second user terminal).

[0105] Here, the message center may query the call status of the first user terminal in the record of the first user terminal. When determining that the call status of the first user terminal is a video call status, the message center may send the information to be displayed to the service server.

[0106] S202: When it is determined based on the call status that the first user terminal is in a video call state, send information to be displayed to a service server so that the service server processes the information to be displayed.

[0107] In an embodiment of the present disclosure, the service server can process the information to be displayed by the following steps: first, extract the text description information of the information to be displayed. Second, determine the call video stream to be displayed on the first user terminal at the next moment. Finally, perform video subtitle synthesis based on the text description information and the call video stream to obtain the target video stream, and send the target video stream to the first user terminal for display.

[0108] Here, after receiving the information to be displayed, the service server can extract the text description information with the display information. The information to be displayed can be video information or text information.

[0109] Here, in the case where the information to be displayed is video information, the service server may extract text information in the video information as text description information, or summarize the content of the video information and use the summarized text content as text description information.

[0110] In the case that the information to be displayed is text information, the service server may use the text information as the text description information of the information to be displayed, or summarize the text and use the summarized text information as the text description information.

[0111] Determine a call video stream sent by a third user terminal that needs to be displayed on the first user terminal at the next moment, wherein the third user terminal is a user terminal currently connected to the first user terminal for a video call.

[0112] In an embodiment of the present disclosure, after determining the text description information, the service server may send a video subtitle synthesis processing request to the media plane (ie, VoNR+media plane) through a capability network element (ie, VoNR+capability network element).

[0113] Here, the video subtitle synthesis processing request carries text description information and the call video stream. After the media plane receives the video subtitle synthesis processing request, it can synthesize the video subtitles based on the text description information and the call video stream to obtain the target video stream.

[0114] Here, the service server may call the interface provided by the capability network element to instruct the media plane to perform video subtitle synthesis based on the text description information and the call video stream.

[0115] In the above implementation, when the user terminal is in a video call state, the text description information of the information to be displayed received in the video call can be extracted. The text description information is synthesized with the call video stream that needs to be displayed on the user terminal at the next moment, so that the information to be displayed is displayed to the user of the user terminal in the form of subtitles. In the case of avoiding the user missing important information, it will not affect the user's current video call connection, thereby improving the user's experience of the video call.

[0116] In an optional implementation, the above step of determining the call status of the first user terminal specifically includes the following steps:

[0117] The call status of the first user terminal is determined based on the status record information of the first user terminal and based on the information content of the status record information.

[0118] In an embodiment of the present disclosure, when a call event subscription is performed through an interface opened by a capability network element, the message center can record the call status of the first user terminal and obtain the status record information of the first user terminal. The call event subscription is used to indicate that the message center can record the status of the first user terminal.

[0119] Here, when the message center determines that the first user terminal receives the information to be displayed sent by the second user terminal, the call status of the first user terminal can be determined based on the information content of the status record information.

[0120] For example, the information content recorded in the status record information is used as the call status of the first user terminal.

[0121] In an optional implementation, based on the above described embodiment, the method further includes the following steps:

[0122] First, obtaining a video call establishment notification sent by the capability network element after detecting a video communication connection between the first user terminal and the third user terminal;

[0123] Then, based on the video call establishment notification, the information content is modified to the video call status.

[0124] In an embodiment of the present disclosure, when it is detected that the first user terminal initiates a video call request to the third user terminal, or that the third user terminal initiates a video call request to the first user terminal, the VoLTE AS (Voiceover Long-Term Evolution Application Server) can generate a call event notification.

[0125] Afterwards, the VoLTE AS can send a call event notification of the first user terminal to the capability network element. The capability network element can send the call notification event to the message center and the service server (ie, service AS) when the message center (ie, 5GMC) subscribes to the call event.

[0126] Here, when the message center receives the above-mentioned call event notification, it can reply to the capability network element with a successful reception signaling, for example, a sip signaling.

[0127] Here, the VoLTE AS can send the call event notification of the first user terminal to the capability network element and send the call event notification to the called terminal at the same time. Wherein, when the first user terminal is the calling terminal, the third user terminal is the called terminal; when the third user terminal is the calling terminal, the first user terminal is the called terminal.

[0128] Afterwards, the called end rings and performs media negotiation with the calling end to determine media parameters for the video call connection, wherein the media parameters include at least one of the following: a call address and a coding method.

[0129] After the VoLTE AS detects that the media parameters are determined and the called terminal is off-hook, it is determined that the video call connection of the first user terminal is successfully established, and a call establishment notification is generated.

[0130] Afterwards, the VoLTE AS may send the call establishment notification to the capability network element. After the capability network element successfully receives the call establishment notification, it may return a successful reception signaling to the VoLTE AS. In addition, the capability network element may send the call establishment notification to the message center and the service server.

[0131] Here, when the message center receives the call establishment notification, the information content of the status record information of the first user terminal can be modified to the video call status.

[0132] In an optional implementation, based on the above described embodiment, the method further includes the following steps:

[0133] First, obtaining a video call interruption notification sent by the capability network element after detecting that the first user terminal interrupts the video call connection;

[0134] Secondly, based on the video call interruption notification, the information content is modified from the video call status to the call idle status.

[0135] In an embodiment of the present disclosure, when the VoLTE AS detects that the first user terminal is on-hook, a video call interruption notification may be generated and sent to a capability network element.

[0136] Here, after receiving the above video call interruption notification, the capability network element may reply to the VoLTE AS to accept the successful signaling (ie, sip signaling), and send the video call interruption notification to the message center and the service server.

[0137] Afterwards, the message center may modify the information content of the status record information of the first user terminal to a call idle state based on the video call interruption notification.

[0138] See also Figure 3As shown, it is an interactive diagram of call establishment processing in a method for displaying information during a call provided by an embodiment of the present disclosure, wherein:

[0139] First, the message center (i.e., 5GMC) subscribes to the call event to the capability network element (i.e., VoNR+ capability network element). Second, the VoLTE AS generates a call event notification when it detects that the first user terminal initiates a video call request to the third user terminal. Second, the VoLTE AS sends the call event notification to the capability network element. Second, the capability network element sends the call event notification to the message center and the service server (i.e., service AS).

[0140] Secondly, the VoLTE AS sends a call event notification to the third user terminal. Secondly, the called party and the calling party perform media negotiation to determine the media parameters for the video call connection. Secondly, after the VoLTE AS detects that the media parameter determination is complete and the called party goes off-hook, a call establishment notification is generated.

[0141] Secondly, the VoLTE AS sends the call establishment notification to the capability network element. Secondly, the capability network element sends the call establishment notification to the message center and the service server. Finally, after receiving the call establishment notification, the message center modifies the information content of the state record information of the first user terminal to the video call state.

[0142] The service server instructs the media plane to complete the audio and video anchoring in the video call and establish two video streams (ie, the video stream sent from the first user terminal to the third user terminal; the video stream sent from the third user terminal to the first user terminal).

[0143] See also Figure 4 As shown, it is an interactive diagram of video subtitle synthesis processing in a method for displaying information during a call provided by an embodiment of the present disclosure, wherein:

[0144] First, the second user terminal sends the information to be displayed to the message center (ie, 5GMC). Secondly, when the message center determines that the first user terminal is in a call state, it sends the information to be displayed to the service server (ie, service AS). Secondly, the service server extracts the text description information of the information to be displayed.

[0145] Secondly, the service server sends a subtitle synthesis request to the capability network element (ie, the VoNR+ capability network element), wherein the subtitle synthesis request carries text description information.

[0146] Secondly, the capability network element instructs the media plane (ie, the VoNR+ media plane) to perform video subtitle synthesis processing based on the text description information and the call video stream.

[0147] Finally, the media plane performs video subtitle synthesis processing on the text description information and the call video stream to obtain the target video stream, and sends the target video stream to the first user terminal for display.

[0148] See also Figure 5 As shown, it is an interactive diagram of an on-hook process in a method for displaying information during a call provided by an embodiment of the present disclosure, wherein:

[0149] First, when the VoLTE AS detects that the first user terminal is on-hook, it generates a video call interruption notification. Second, the VoLTE AS sends the video call interruption notification to the capability network element.

[0150] Secondly, the capability network element (ie, the VoNR+ capability network element) sends the video call interruption notification to the message center and the service server. Finally, the message center modifies the information content of the state record information of the first user terminal to the call idle state based on the video call interruption notification.

[0151] Based on the same inventive concept, the embodiments of the present disclosure also provide a system for displaying information during a call corresponding to the method for displaying information during a call. Since the principle of solving the problem by the system in the embodiments of the present disclosure is similar to the method for displaying information during a call in the embodiments of the present disclosure, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.

[0152] Reference Figure 6 As shown, it is an architecture diagram of a system for displaying information during a call provided by an embodiment of the present disclosure, and the system includes: a service server (i.e., service AS), a capability network element (i.e., VoNR+capability network element), a media plane (i.e., VoNR+media plane), a long-term evolution voice bearer service server (i.e., VoLTE AS), and a message center (i.e., 5GMC).

[0153] The service server, capability network element, media plane, long term evolution voice bearer service server and message center are all in communication connection.

[0154] Among them, capability network elements are mainly used for:

[0155] 1) Access management for business servers.

[0156] 2) Provide atomic operation capabilities on the media plane to the business server, including: media stream replication, subtitle stream synthesis, virtual background replacement (including background blur), virtual avatar replacement, media element synthesis (static image or sequence, no follow-up gestures), playback, number collection, subtitle prompts, etc.

[0157] 3) Based on the need of the service server to subscribe to the call event from the VoLTE AS, the subscription events required by different service ASs are merged.

[0158] 4) Report call-related information to the corresponding service AS based on the user's service subscription.

[0159] 5) Supports storing user subscription information from the new call operation management platform as transparent data in HSS (Home Subscriber Server) / UDM (User Data Management) and supports functions such as downloading, updating, subscribing, and updating notification of this data.

[0160] 6) Support the selection of the unified entry address of the media capability platform based on user location information.

[0161] 7) Network element role that carries intelligent network capabilities.

[0162] The media side is used for:

[0163] 1) Receive control from the VoLTE AS to create, update, and delete media resources. Control is performed before and after a call is established.

[0164] 2) Accept instructions and control from capability network elements to perform basic audio and video capabilities such as media stream replication, number collection, subtitle prompts, and playback.

[0165] 3) Accept instructions and control from the capability network element and perform related media synthesis: subtitle stream synthesis, virtual background replacement (including background blur), virtual avatar replacement, media element synthesis (static image or sequence, no follow-up gesture).

[0166] 4) For audio and video content recognition with changing requirements, which is not a part of high real-time video processing requirements, an external media capability platform is required to implement it; and the media capability platform notifies the industry server to implement the mapping of the recognized content to the superimposed model, and then downloads the model to the media surface to implement the synthesis action.

[0167] 5) Download and cache of media material resources: For general materials, it supports subscribing to media resources from the new call operation management platform and receiving general materials issued by the new call operation management platform; for materials that do not exist locally (personalized materials or general materials), it supports real-time downloading from the new call operation management platform; the media side needs to cache general materials and personalized materials.

[0168] 6) When releasing audio and video media resources, if there is an ongoing media copy task, a stop media copy message needs to be sent to the media capability platform.

[0169] Business servers are used to:

[0170] The service AS includes the intelligent translation service AS and the fun call service AS. The functions provided by the intelligent translation service AS are: subtitle transcription, subtitle translation, display startup, and service settings; the functions provided by the fun call service AS are: background replacement, virtual avatar replacement, voice expression rain, gesture effects, and service settings. The application server needs to interact with the capability network element to implement the service. When registering with the service server, it needs to carry the device-level subscription to indicate the events that the capability network element needs to report; the application server needs to support docking and instruct the media capability platform to perform the media processing required for the service.

[0171] Long-term evolution voice bearer service server:

[0172] 1) The VoLTE AS needs to report call events and call parameter information based on the capability network element user-level subscription.

[0173] 2) The VoLTE AS needs to map the abcd in the sbc-domain parameter (sbc-domain = sbc.abcd.xyz.chinamobile.com, where abcd is the area code derived from TAI or E-CGI) carried by the SBC (Session Border Controller) in the PANI (P-Access-Network-Info) header field of the call initiation request or response message to the visited VoNR+ media plane address (which can be addressed based on the domain name).

[0174] 3) The VoLTE AS needs to support storing the user-level subscription data of VoNR+ capable network elements in HSS / UDM in a transparent data manner.

[0175] The Message Center is used to:

[0176] The message center has the ability to process short messages and basic multimedia messages, and adds a new network interface with MaaP (Massage as a platform) to access new MaaP industry users. At the same time, it retains the same network interface as the short message center on the network interface side, supports existing SP (SMS definition) and SMS applications, and provides 5G message sending and receiving functions for terminals that support 5G messages.

[0177] Based on the same inventive concept, an embodiment of the present disclosure also provides a device for displaying information during a call corresponding to the method for displaying information during a call. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the method for displaying information during a call in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0178] Reference Figure 7FIG. 1 is a schematic diagram of a device for displaying information during a call provided by an embodiment of the present disclosure, wherein the device comprises: an extraction module 11, a first determination module 12, and a synthesis module 13, wherein:

[0179] an extraction module, configured to extract text description information of the information to be displayed if the information to be displayed of the first user terminal is detected when the first user terminal is in a video call state; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal;

[0180] A first determining module, used to determine a call video stream to be displayed on the first user terminal at a next moment;

[0181] A synthesis module is used to synthesize video subtitles based on the text description information and the call video stream to obtain a target video stream, and send the target video stream to the first user terminal for display.

[0182] Reference Figure 8 FIG. 2 is a schematic diagram of another device for displaying information during a call provided by an embodiment of the present disclosure, wherein the device comprises: a second determining module 21, a sending module 22, wherein:

[0183] A second determination module is used to determine the call status of the first user terminal when the information to be displayed of the first user terminal is detected; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal;

[0184] The sending module is used to send the information to be displayed to the service server when it is determined based on the call status that the first user terminal is in a video call state, so that the service server processes the information to be displayed.

[0185] In the embodiment of the present disclosure, when the user terminal is in a video call state, text description information of the information to be displayed received in the video call can be extracted. The text description information is synthesized with the call video stream that needs to be displayed on the user terminal at the next moment, so that the information to be displayed is displayed to the user of the user terminal in the form of subtitles. While avoiding the user from missing important information, it will not affect the user's current video call connection, thereby improving the user's experience of the video call.

[0186] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0187] Corresponds to Figure 1 or Figure 2The present disclosure also provides an electronic device 900, such as Fig. 9 FIG. 1 is a schematic diagram of the structure of an electronic device 900 provided in an embodiment of the present disclosure, including:

[0188] Processor 91, memory 92, and bus 93; memory 92 is used to store execution instructions, including internal memory 921 and external memory 922; the internal memory 921 is also called internal memory, which is used to temporarily store the operation data in the processor 91 and the data exchanged with the external memory 922 such as a hard disk. The processor 91 exchanges data with the external memory 922 through the internal memory 921. When the electronic device 900 is running, the processor 91 communicates with the memory 92 through the bus 93, so that the processor 91 executes the following instructions:

[0189] If the first user terminal is in a video call state and the information to be displayed of the first user terminal is detected, the text description information of the information to be displayed is extracted; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal;

[0190] Determining a call video stream to be displayed on the first user terminal at a next moment;

[0191] Video subtitles are synthesized based on the text description information and the call video stream to obtain a target video stream, and the target video stream is sent to the first user terminal for display.

[0192] The processor 91 may also execute the following instructions:

[0193] In the case of detecting information to be displayed of the first user terminal, determining the call state of the first user terminal; wherein the information to be displayed is information to be displayed sent by the second user terminal to the first user terminal;

[0194] In a case where it is determined based on the call status that the first user terminal is in a video call state, the information to be displayed is sent to a service server.

[0195] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0196] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0197] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0198] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0199] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0200] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0201] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for displaying information during a call, characterized in that: Applied to business servers, including: If the first user terminal is in a video call state and the information to be displayed of the first user terminal is detected, the text description information of the information to be displayed is extracted; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal; Determining a call video stream to be displayed on the first user terminal at a next moment; Video subtitles are synthesized based on the text description information and the call video stream to obtain a target video stream, and the target video stream is sent to the first user terminal for display.

2. The method according to claim 1, characterized in that The synthesizing video subtitles based on the text description information and the call video stream to obtain a target video stream includes: Determining whether the first user terminal meets the video subtitle synthesis condition; When the first user terminal meets the video subtitle synthesis condition, the text description information and the call video stream are synthesized into video subtitles to obtain a target video stream.

3. The method according to claim 1 or 2, characterized in that The synthesizing video subtitles based on the text description information and the call video stream to obtain a target video stream includes: Determine the text attributes of the text description information; wherein the text attributes include but are not limited to: the fusion special effect attribute of the text description information and the real-time attribute of the text description information; Based on the text attributes, the text description information, the call video stream and the special effect fusion data of the text description information are synthesized into video subtitles to obtain the target video stream.

4. The method according to claim 3, characterized in that The step of synthesizing video subtitles by combining the text description information, the call video stream, and the special effect fusion data of the text description information based on the text attribute to obtain the target video stream includes: A video subtitle fusion application is sent to the capability network element, so that the capability network element synthesizes the video subtitles with the text description information, the call video stream and the special effect fusion data of the text description information to obtain the target video stream when the text attribute meets the attribute requirements.

5. The method according to claim 1, characterized in that If the first user terminal is in a video call state, the information to be displayed of the first user terminal is detected, and the text description information of the information to be displayed is extracted, including: If the first user terminal is in a video call state and the to-be-displayed information of the first user terminal sent by the message center is detected, the text description information of the to-be-displayed information is extracted.

6. A method for displaying a message during a call, characterized in that: Applicable to the message center, including: In the case of detecting information to be displayed of the first user terminal, determining the call state of the first user terminal; wherein the information to be displayed is information to be displayed sent by the second user terminal to the first user terminal; When it is determined based on the call status that the first user terminal is in a video call state, the information to be displayed is sent to a service server, so that the service server processes the information to be displayed according to the method described in any one of claims 1 to 5 above.

7. The method according to claim 6, characterized in that The determining the call status of the first user terminal includes: The call status of the first user terminal is determined based on the status record information of the first user terminal and based on the information content of the status record information.

8. The method according to claim 7, characterized in that The method further comprises: Acquiring a video call establishment notification sent by the capability network element after detecting a video communication connection between the first user terminal and the third user terminal; The information content is modified to a video call status based on the video call establishment notification.

9. The method according to claim 8, characterized in that The method further comprises: Acquiring a video call interruption notification sent by the capability network element after detecting that the first user terminal interrupts the video call connection; The information content is modified from the video call state to a call idle state based on the video call interruption notification.

10. A device for displaying information during a call, characterized in that: include: an extraction module, configured to extract text description information of the information to be displayed if the information to be displayed of the first user terminal is detected when the first user terminal is in a video call state; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal; A first determining module, used to determine a call video stream to be displayed on the first user terminal at a next moment; A synthesis module is used to synthesize video subtitles based on the text description information and the call video stream to obtain a target video stream, and send the target video stream to the first user terminal for display.

11. A device for displaying information during a call, characterized in that: include: A second determination module is used to determine the call status of the first user terminal when the information to be displayed of the first user terminal is detected; wherein the information to be displayed is the information to be displayed sent by the second user terminal to the first user terminal; A sending module is used to send the information to be displayed to the service server when it is determined that the first user terminal is in a video call state based on the call state, so that the service server processes the information to be displayed according to the method described in any one of claims 1 to 5 above.

12. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 9 are implemented.

Images