Conversation interface interaction method and related device

Abstract

An interaction method of a call interface is applied to an electronic device which carries an AI assistant and is provided with a display screen, and the method comprises the steps that it is detected that an AI call mode is started, and the AI call mode refers to a mode that the AI assistant is applied to make a call with an incoming call object; and detecting a first operation, the first operation being used for indicating to switch an AI call mode to an artificial call mode, and the artificial call mode being a mode in which a user communicates with the incoming call object. According to the method, switching between the user call mode and the AI call mode is supported, interaction with the AI assistant through an interface is supported in the AI call mode process, the AI assistant is guided to better communicate with the incoming call object, and therefore the user experience and the working efficiency are improved.

Description

Technical Field

[0001] The present application relates to the technical field of terminal equipment, and in particular to an interactive method and related device for a call interface. Background Art

[0002] With the development of society and the revolution of manufacturing technology, the world is accelerating into a digital intelligent society where everything is sensed, connected and intelligent, and the digitization of electronic devices has gradually become the mainstream. In the context of the digital age, people have higher requirements for efficient work and life. For example, in multi-tasking scenarios such as online meetings and important phone calls, users can use AI assistants to help answer calls and record important information during calls to improve efficiency.

[0003] At present, using AI assistants to answer calls has become a mainstream scenario, but due to the complexity of call scenarios and the diversity of call types, AI assistants find it difficult to handle complex and changeable call scenarios well. Therefore, a call method that supports user interaction with AI assistants is essential. Summary of the invention

[0004] This application provides a method for call interface interaction, which can help users answer calls and generate key information summary cards, thereby improving user experience and work efficiency.

[0005] The first aspect of the present application provides an interactive method for a call interface. The method is applied to an electronic device with an AI assistant application, and the electronic device is not limited to smart terminal electronic devices such as mobile phones, computers, tablets, and watches. The method includes: detecting that the AI ​​call mode is enabled, and the AI ​​call mode refers to a mode in which the AI ​​assistant is used to talk to the callee; detecting a first operation, and the first operation is used to indicate that the AI ​​call mode is switched to a manual call mode, and the manual call mode refers to a mode in which the user talks to the callee. The method helps users receive some key call information by using an AI assistant to answer the call, and realizes interaction during the call through the interaction between the AI ​​call mode and the manual mode, thereby reducing the AI ​​assistant's incorrect response to the call content and solving the problem that the AI ​​assistant is difficult to deal with complex call scenarios, which can improve the user experience and work efficiency.

[0006] In a possible implementation, the method further includes: detecting input of a prompt word, generating a call voice signal based on the prompt word, and sending the call voice signal to the caller.

[0007] In this solution, users can set the AI ​​assistant's reply content and method based on the prompt words that pop up on the interface, so that the AI ​​assistant can adjust its status and content in time according to the needs of the actual scene, thus providing a better user experience.

[0008] In a possible implementation, the method further includes: the call interface displays an information card and a summary card, wherein the information card is used to indicate key content text in the call information, and the summary card is used to indicate summary content text in the call information.

[0009] In this solution, the call interface will display information cards and summary cards to record the content of the call, thereby improving the user experience and work efficiency.

[0010] In a possible implementation, the method further includes: the call interface displays an AI assistant icon, and the AI ​​assistant icon is used to interact with the user to implement mode switching.

[0011] In this solution, an interface interaction scenario is proposed, in which users switch call modes by operating the AI ​​assistant icon.

[0012] In a possible implementation, the prompt words specifically include fixed recommendation words, and the fixed recommendation words include tone recommendation words and control recommendation words; the tone recommendation words are used to indicate the tone of the output voice text; the control recommendation words are used to output voice text to control the progress and duration of the call.

[0013] In this solution, prompt words include tone recommendations and control recommendations. Tone recommendations can help the AI ​​assistant adapt to the current conversation context by adjusting its tone by selecting tone types, including cute tone, lively tone, serious tone, gentle tone, angry tone, etc., and better cope with expressing user emotions in complex scenarios; control recommendations are mainly used to control and remind call duration through call language selection.

[0014] In a possible implementation, the prompt word specifically includes a real-time recommendation word, which is text data generated according to real-time call content and is used to reply to the caller.

[0015] In this solution, a real-time recommendation word is proposed. The real-time recommendation word is mainly generated based on the call content with the caller to answer some key information prompt words of the caller. For example, the caller asks which restaurant to go to, Restaurant A or Restaurant B? The call interface can show the options of "Restaurant A" and "Restaurant B", or the caller proposes the time to book a meeting room. The call interface provides options based on the choice given by the user for selection. This method provides users with good interaction in AI calls, which can help AI assistants better express users' ideas and wishes, thereby effectively improving users' usage experience and work efficiency.

[0016] In a possible implementation, the prompt word specifically includes user input text, where the user input text is text data manually input by the user and used to express the user's wishes and ideas.

[0017] In this solution, a method is proposed that supports users to manually input text and express their ideas in real time to interact with the caller. This can help AI assistants better deal with complex scenarios, better express the user's wishes, and in some scenarios can be more tactful and humble, which is more conducive to maintaining interpersonal relationships.

[0018] In one possible implementation, the prompt word specifically includes user input voice, and the method includes: detecting a second operation, the second operation being used to indicate receiving the user input voice; converting the user input voice into an AI assistant voice; and sending the AI ​​assistant voice to the caller.

[0019] In this solution, an AI assistant is proposed to answer calls, and humans help the AI ​​assistant to better answer the caller's questions by inputting voice.

[0020] In a possible implementation, the method further includes: when a third operation is detected, adjusting the conversation speed of the AI ​​assistant.

[0021] In this solution, a method for adjusting the speech speed of an AI assistant is proposed, which can enhance the user experience of using the AI ​​assistant and enable the AI ​​assistant to better simulate human behavior and language rhythm.

[0022] In a possible implementation, the call interface displays an information card and a summary card, wherein the information card is used to indicate key content text in the call information, and the summary card is used to indicate summary content text in the call information.

[0023] In one possible implementation, the call interface displays information cards and summary cards specifically including: detecting an audio signal, identifying information of a preset type in the audio signal to obtain key content text and summary text; generating an information card based on the key content text; and generating a summary card based on the summary text.

[0024] In this solution, users can identify key information, preset information and other information based on the audio information of the caller, and based on this information, they can extract important summary information and generate summary cards; and they can summarize the information in the call content to generate a summary text for recording and reminding users.

[0025] In a possible implementation, the preset types in the audio signal include but are not limited to telephone numbers, addresses, and schedules.

[0026] In a possible implementation, the summary text is text content that summarizes the call content within a preset time period.

[0027] In this solution, the AI ​​assistant can generate content summary text based on the large language model according to the audio call content within a preset duration.

[0028] A second aspect of the present application provides an electronic device of an AI assistant, the device comprising: a detection module for detecting the activation of an AI call mode, the AI ​​call mode referring to a mode of using an AI assistant to talk to a callee; the detection module is also used to detect a first operation, the first operation being used to indicate switching the AI ​​call mode to a manual call mode, the manual call mode referring to a mode of a user talking to a callee.

[0029] In a possible implementation, the device further includes: an input module, configured to input a prompt word, generate a call voice signal based on the prompt word, and send the call voice signal to the caller.

[0030] In a possible implementation, the device further includes: a display module, configured to display information cards and summary cards, wherein the information cards are configured to indicate key content text in the call information, and the summary cards are configured to indicate summary content text in the call information.

[0031] In a possible implementation, the display module is also used to display an AI assistant icon, and the AI ​​assistant icon is used to interact with the user to implement mode switching.

[0032] In a possible implementation, the prompt words specifically include fixed recommendation words, and the fixed recommendation words include tone recommendation words and control recommendation words; the tone recommendation words are used to indicate the tone of the output voice text; the control recommendation words are used to output voice text to control the progress and duration of the call.

[0033] In a possible implementation, the prompt word specifically includes a real-time recommendation word, which is text data generated according to real-time call content and is used to reply to the caller.

[0034] In a possible implementation, the prompt word specifically includes user input text, where the user input text is text data manually input by the user and used to express the user's wishes and ideas.

[0035] In one possible implementation, the prompt word specifically includes user input voice, and the method includes: detecting a second operation, the second operation being used to indicate receiving the user input voice; converting the user input voice into an AI assistant voice; and sending the AI ​​assistant voice to the caller.

[0036] In a possible implementation, the detection module is further used to detect a third operation, where the third operation is used to instruct adjustment of the conversation speed of the AI ​​assistant.

[0037] In a possible implementation, the display module is used to display information cards and summary cards, wherein the information cards are used to indicate key content text in the call information, and the summary cards are used to indicate summary content text in the call information.

[0038] In a possible implementation, the device also includes a text generation module, which is used to generate text data from the detected audio signal, specifically including: the detection module detects the audio signal, identifies the preset type of information in the audio signal to obtain key content text and summary text; the text generation module generates an information card based on the key content text; the text generation module generates a summary card based on the summary text.

[0039] In a possible implementation, the preset types in the audio signal include but are not limited to telephone numbers, addresses, and schedules.

[0040] In a possible implementation, the summary text is text content that summarizes the call content within a preset time period.

[0041] The third aspect of the present application provides an electronic device of an AI assistant, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method of the above-mentioned first aspect or any implementation of the first aspect is implemented. For the steps in each possible implementation of the first aspect executed by the processor, please refer to the first aspect for details, and no further description is given here.

[0042] The fourth aspect of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method of any implementation manner of the first aspect.

[0043] The fifth aspect of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method implemented in any one of the first aspects.

[0044] The beneficial effects of the second to fifth aspects mentioned above can be referred to the introduction of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a call answering interface provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of starting an AI call mode provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of generating and typing a call voice signal based on a prompt word provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a call interface after calling out a virtual keyboard provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of a call interface after calling out a virtual keyboard provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of generating a summary card provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of entering a call card interface from an answering interface provided in an embodiment of the present application;

[0053] Fig. 9 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those of ordinary skill in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0055] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchanged where appropriate, so that the embodiments can be implemented in a sequence other than that illustrated or described in the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps that appear in the present application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0056] The division of units in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units, and some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0057] To facilitate understanding, some technical terms involved in the embodiments of the present application are first introduced below.

[0058] (1) AI Assistant

[0059] AI Assistant is an artificial intelligence assistant with functions such as AI question-and-answer assistant, AI drawing assistant, AI analysis assistant, and intelligent voice translation robot, providing users with an efficient and convenient artificial intelligence system platform.

[0060] (2) Message Card

[0061] There are many ways to classify message cards. For example, for WeChat, which is an instant messaging software, unread messages can be classified into text messages, picture messages, video messages, voice messages, and voice / video call messages according to the content; and can be divided into personal messages and group messages according to the attributes of the sender. These different types can be used as message card categories.

[0062] (3) Large Language Model

[0063] Large language models are one of the applications of deep learning, especially in the field of natural language processing (NLP). Large language models need to be trained on a large amount of text data to learn various patterns and structures of language. The goal of these models is to understand and generate human language in order to conduct effective conversations and answer various questions.

[0064] At present, with the advancement of technology and the development of society, telephone conferences in daily work have become overwhelmed, and important calls are often missed due to the multitude of things. In order to make people's lives more convenient and work more efficient, answering calls with the help of AI assistants is an important means. AI assistants can communicate with callers according to the preset tone, and can also generate a call summary for the key information in the conversation content, so that even if the user is not there, he can still clearly understand the content of the call.

[0065] This method can effectively avoid missing important calls by using an AI assistant to answer calls; by supporting the switching between AI mode and manual mode, it can effectively avoid the problem that the AI ​​assistant has difficulty in coping with complex scenarios and processing complex call content; the method also supports users to guide the tone and content of AI calls in AI call mode, so that the AI ​​assistant can better handle the call content and answer questions in the call according to the user's wishes; in addition, the summary information generated by the large language model based on the call audio data can be used to obtain key information of the conference call and the summary information can be used to record the call content within the preset call duration, which can effectively improve work efficiency and user experience.

[0066] In view of this, the embodiment of the present application provides an interactive method for a call interface, which allows the use of an AI assistant to answer important calls based on the connection settings for important calls. When the missed call rings for a period of time, the AI ​​assistant automatically answers the call and generates a key information summary card based on the call content. In addition, during the AI ​​assistant's answering process, the user can intervene at any time to interact with the AI ​​assistant and guide the AI ​​assistant to make a better call. In addition, during the call, the AI ​​assistant function can be turned off at any time and the call can be continued using the manual answering method.

[0067] See also Figure 1 , Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1As shown, the application scenario includes user 001 and electronic device 002. Among them, electronic device 002 can be a smart terminal electronic device such as a mobile phone, tablet, laptop, watch, etc. Specifically, user 001: is usually the owner of electronic device 002, who obtains network information and communicates with others through electronic device 002. Electronic device 002: is equipped with an operating system and installed with various software, such as call applications, text messages, instant messaging software, email applications, etc. The electronic device obtains satellite / network call signals through communication modules such as antennas, activates call applications, and displays call interfaces.

[0068] In one possible implementation, user 001 sets the AI ​​assistant in electronic device 002 to automatically answer the call after the call rings for a period of time (the time can be preset, assuming it is set to 40s). For example, assuming that the electronic device rings for 40 seconds and the user does not answer the call, the AI ​​assistant automatically answers the call.

[0069] In one possible implementation, user 001 receives a call from "Zhang San". At this time, the answer and hang up buttons are displayed on the interface of electronic device 002, and a "swipe up to answer by AI assistant" icon is displayed at the bottom. By swiping up the AI ​​assistant button, the AI ​​assistant can answer the call.

[0070] The above scheme is explained by taking the AI ​​assistant answering the phone as an example. In other implementations of the present application, the AI ​​assistant can also help answer voice calls, etc., and can use voice communication with other users.

[0071] See also Figure 2 , Figure 2 A schematic diagram of a call answering interface provided in an embodiment of the present application.

[0072] S101: AI assistant icon is displayed on the incoming call interface

[0073] The electronic device detects an incoming call signal and displays an incoming call interface with an AI assistant icon. A call application is installed on the electronic device, and the call application can be a satellite call application that comes with the system or a network call application. The call application receives the incoming call signal through the communication module, such as receiving the satellite signal through the antenna. After receiving the satellite signal, the call application is started and displayed on the front, showing the call interface.

[0074] In this solution, the incoming call interface includes the pre-answering interface and the answering interface. The pre-answering interface is the call application interface before the user performs the preset operation, and the answering interface is the interface after the user performs the preset operation and the call is connected. The preset operation is the system-defined operation for answering the call, for example, dragging the answer button to move through gestures to answer the call.

[0075] The AI ​​assistant icon is displayed on both the pre-answering interface and the answering interface. In this solution, the preset operation can also be an interactive operation with the AI ​​assistant, such as dragging the AI ​​assistant icon to move, interacting with the AI ​​assistant through voice input, etc.

[0076] For example, Figure 2 The pre-answering interface and the answering interface are shown respectively. Figure 2 a and Figure 2 b is the call application interface for manual answering. Figure 2 c is the call application interface answered by AI. Figure 2 a is the interface before answering the call, Figure 2 b and Figure 2 c is the answering interface. Figure 2 In the pre-answering interface of a, in addition to the answer button and hang up button, Figure 1 In addition to the hang-up button, the AI ​​assistant icon is also displayed at the bottom of the screen in the answering interface of b. Before the user interacts with the AI ​​assistant icon, only part of the AI ​​assistant icon is displayed on the screen, and part of it is off the screen, that is, the AI ​​assistant is not fully displayed in the interface before answering the call. The user drags the AI ​​assistant icon to move it, and the AI ​​assistant icon can be fully displayed on the screen. Figure 2 c, in AI answering mode, the AI ​​assistant icon is fully displayed at the preset position on the screen, and the AI ​​assistant application will synchronously receive the audio signal, understand the audio, and then combine the user's input and the AI ​​assistant's semantic understanding of audio-to-text to output AI voice and send it to the caller.

[0077] S102: Enable AI call mode

[0078] When the AI ​​assistant icon is detected to move from the bottom of the screen to the middle of the screen, the AI ​​call mode is enabled and the call audio is synchronously transmitted to the AI ​​assistant application; in the pre-answering interface, the user can interact with any of the answer, hang up, and AI assistant icons to handle the call. The interaction of the answer and hang up buttons is consistent with the existing technology. The user can drag the AI ​​assistant icon to move, drag the AI ​​assistant icon from the bottom of the screen to the preset area in the middle of the screen, or drag the distance greater than the preset value, then the AI ​​call mode is enabled (the opposite of the AI ​​call mode is the manual answering mode).

[0079] See also Figure 3 , Figure 3 A schematic diagram of starting an AI call mode provided in an embodiment of the present application. For example, Figure 3The process of enabling the AI ​​call mode is shown. After the user selects the AI ​​assistant icon, he drags the AI ​​assistant to move on the screen. When it moves to the preset area of ​​the screen (such as the dotted box area in the middle picture), if the gesture is released at this time, it is considered that the conditions for answering the AI ​​call are met, and the AI ​​call is enabled. After the release, the AI ​​assistant icon will move from the gesture release position to the preset position of the answering interface.

[0080] In one possible implementation, in AI call mode, in addition to sending the received audio data to the call application, the electronic device will also send it to the AI ​​assistant application simultaneously. After the AI ​​assistant calls the language model based on the received audio data for understanding, it outputs the reply text corresponding to the audio, and converts the reply text into a voice signal and sends it to the call application, which is then sent by the call application to the caller. At the same time, in AI call mode, when the call application sends a voice signal to the caller, it will also synchronously send the voice signal to the audio output module for output, such as directly through the speaker, or through the Bluetooth module to the Bluetooth headset, so that the user can obtain all conversations between the AI ​​assistant and the caller in real time as a third party.

[0081] S103: Generate and enter a call voice signal based on the prompt word

[0082] The input of the prompt word is detected, a call voice signal is generated based on the prompt word, and the call voice signal is sent to the caller; in the AI ​​call answering mode, the user can enter the prompt word, and the AI ​​assistant will generate a reply text based on the prompt word entered by the user and its understanding of the received audio information, and convert the reply text into call voice, send the call voice to the call application, and then the call application will send it to the caller / audio output module.

[0083] In one possible implementation, the prompt word can be input by a recommended word selected by the user in the call interface, or by an input sentence entered by the user calling a virtual keyboard in the call interface. Specifically, after the AI ​​call answering mode is enabled, the AI ​​assistant can display recommended words in the call interface, and the user can influence the call sentence output by the AI ​​by selecting the recommended words. In addition, the call interface in the AI ​​call mode can also display a virtual keyboard icon. The user clicks the virtual keyboard icon to enable the virtual keyboard for input and directly enters a sentence as the prompt word.

[0084] In this solution, there are also the following ways to generate recommendation words:

[0085] Recommended words can include fixed recommended words and real-time recommended words, where the recommended words are fixed words applicable to all AI calls. Real-time recommended words are generated by the AI ​​assistant based on the understanding of the received voice signal and are directly associated with the audio sentence of the caller. In other words, real-time recommended words are generated based on real-time recognition of the current call content. This type of recommended words is generated based on the semantics of the current caller.

[0086] 1) Fixed recommendation words

[0087] Fixed recommended words include tone recommended words and process control recommended words. Tone recommended words are used to influence the wording of the output voice text. For example, if you choose a formal tone or a lively tone, the final output text will be different. The text corresponding to a stern tone is more written, while the text corresponding to a lively tone is more spoken.

[0088] The recommended words for process control are used to control the progress and duration of the call, including quick ending and maintaining the call. For example, if you select quick ending, the AI ​​assistant will add a sentence indicating the expected end of the call after completing the current output / while executing the current output, so as to convey the intention to end the conversation to the caller (rather than hanging up the phone directly).

[0089] 2) Real-time recommended words

[0090] The real-time recommendation words can be generated by the AI ​​assistant application based on the received audio signal, or can be generated by the AI ​​assistant application based on the expected reply content output for the received audio signal. The expected reply content is the text generated by the AI ​​assistant based on the received audio signal and has not yet been sent to the call application.

[0091] Specifically, the real-time recommendation words can be keywords extracted from the text obtained after the received audio signal is converted from speech to text. For example, if the text obtained by speech conversion is "Where to eat tonight, restaurant A or restaurant B?", then the keywords "restaurant A" and "restaurant B" are extracted as recommendation words. In addition, derivative words of keywords can be obtained according to preset rules based on keywords and context. For example, the aforementioned speech is an interrogative sentence, and two choices are given - restaurant A and restaurant B. In addition to choosing an answer from restaurant A and restaurant B, the other party can also decide. Therefore, the derivative word "decided by the other party" can be obtained.

[0092] Alternatively, the real-time recommendation words can also be keywords extracted from the expected reply content, or derived words obtained from the keywords extracted from the expected reply content based on preset rules. For example, if the text obtained by voice conversion is "Do you want to have dinner tonight?", the AI ​​can output the reply words "Agree" and "Reject" as well as "Watch it later", thus obtaining the derived word "Watch it later".

[0093] 3) Presentation of recommended words

[0094] Recommended words can be presented by recommended word type or by the dimension of a single recommended word. Each recommended word corresponds to a bubble, which is presented in a preset area of ​​the call interface. Recommended word bubbles can be presented statically or in a dynamic effect. For example, recommended word bubbles can move within a preset area, for example, a dynamic effect of moving bubbles from inside the screen to outside the screen is presented, and a single recommended word can be presented for a period of time and then for a period of time.

[0095] All recommended words can be displayed on the call interface, or only preset types of recommended words can be displayed on the call interface. Non-preset types of recommended words are folded, and only after the user performs a preset operation, the recommended words of that type are expanded and displayed on the call interface.

[0096] Real-time recommendations have a direct impact on the output results of AI calls, while tone recommendations only modify the output results. Process control recommendations are only called when the user needs them. Therefore, all real-time recommendations are displayed in the call interface in the form of recommendation bubbles. Tone recommendations and process control recommendations only display two type controls. Click the type control to display the recommendation of that type on the call interface.

[0097] See also Figure 4 , Figure 4 A schematic diagram of generating and typing a call voice signal based on a prompt word provided in an embodiment of the present application. For example, Figure 4 a and Figure 4 b shows two different presentation methods. Figure 4 In a, two lines of recommendations are displayed in a static way. One line of real-time recommendations, such as "decline politely", "accept the other party's invitation", "give conclusion later", etc. These real-time recommendations are related to the current call content and need to be recommended in real time according to the current call content; one line of fixed recommendations, such as "relaxed tone", "lively tone", "formal tone", this type of guidance and suggestion for the tone of the AI ​​assistant's reply. All types of recommendations are folded and displayed directly in the form of recommendation bubbles (limited by the screen area, some bubbles are displayed outside the screen). Swipe left and right to display more recommendation bubbles. Figure 4In b, real-time recommended words are displayed around the AI ​​assistant icon and change their position dynamically over time, with dynamic effects (for example, bubbles move from inside the screen to outside the screen and gradually appear with decreasing transparency). Fixed recommended words only display category controls at fixed positions - "tone selection" and "call control". Click the corresponding category control to display the recommended words of that type in the call interface in the form of recommended word bubbles. The results after display refer to Figure 4 Fixed recommendation words in a.

[0098] In one implementation, if the displayed recommended words do not meet the user's needs, the user can also click the virtual keyboard icon in the lower left corner to call out the virtual keyboard and manually enter a sentence as the recommended word.

[0099] See also Figure 5 , Figure 5 A schematic diagram of a call interface after calling out a virtual keyboard is provided in an embodiment of the present application. For example, Figure 5 As shown, after the user enters the custom text and clicks send, the virtual keyboard transmits the text entered by the user to the AI ​​assistant application. After receiving the input text, the AI ​​assistant application uses the input text as a recommended word to generate a reply text. For example, the input text is directly used as the reply text.

[0100] After receiving the recommendation word, the AI ​​assistant application inputs the text converted based on the recommendation word and the received audio into the built-in / cloud language model to obtain the reply text, and converts the reply text into audio and outputs it to the call application.

[0101] When the user selects a tone recommendation word, he can also select different voices to synthesize the audio according to the tone recommendation words when converting the reply text into audio. The correspondence between the tone recommendation words and the voices can be pre-defined.

[0102] For example, different tone recommendation words may correspond to different voice models, and the corresponding voice model is selected according to the tone recommendation words; or different tone recommendation words may correspond to different voice tags, and the voice model is selected based on the tags. It can be understood that in order to output audio uniformity in sound, different voice models can be obtained by fine-tuning the same reference voice to obtain voice models of different tones.

[0103] When the call application sends the received audio, it will also synchronously call the audio output module to play the received audio, so that the user can receive the voice conversation between the AI ​​assistant and the caller in real time.

[0104] In the AI ​​call answering interface, in addition to the recommended word interaction, the following interactions can also be included:

[0105] ① Long press the AI ​​assistant to send voice: This can be achieved in two ways. One is to long press to trigger the AI ​​assistant to receive the voice signal and send the voice signal to the call application, which then sends it to the other end. The other is to long press the AI ​​assistant to trigger the AI ​​call mode interruption and temporarily switch to the manual answering mode. The call application directly receives and sends the user's voice. When it detects that the long press gesture is released, it switches back to the AI ​​call mode.

[0106] ② Drag the AI ​​assistant to move, and adjust the speaking speed of the AI ​​assistant by controlling the direction of movement and giving examples. For example, drag the AI ​​assistant to the left / right to adjust the speaking speed. Of course, it can also be moved up and down. You can adjust the speaking speed by directly dragging the AI ​​assistant to move, or you can activate the speaking speed adjustment function through preset gestures (such as long pressing for more than the preset value), and then adjust the speaking speed by dragging the AI ​​assistant to move.

[0107] S104: AI call mode switched to manual answering mode

[0108] It is detected that the AI ​​assistant icon moves from the middle of the screen to the bottom of the screen, and the AI ​​call mode is switched to the manual answering mode. During the AI ​​call, the user can switch the AI ​​call mode to the manual answering mode by moving the AI ​​assistant icon to the bottom of the screen. Specifically, if the electronic device detects the movement of the AI ​​assistant icon again, the electronic device obtains the position of the AI ​​assistant icon in real time according to the operation gesture. If the AI ​​assistant icon at least partially falls in the preset area (such as outside the screen display area), the electronic device ends the AI ​​call mode, and the call application no longer sends audio data to the AI ​​assistant application, but switches to the manual answering mode and outputs only through the audio output module.

[0109] See also Figure 6 , Figure 6 A schematic diagram of a call interface after calling out a virtual keyboard is provided in an embodiment of the present application. For example, Figure 6 The diagram shows the interactive process of switching from AI call mode to manual answering mode. After selecting the AI ​​assistant icon, drag the AI ​​assistant icon downward and move the AI ​​assistant icon at least partially outside the screen display area to switch to manual answering mode.

[0110] In one possible implementation, the AI ​​assistant application can also generate summary cards and information cards during a call, where the summary card is used to record key information during the call; the information card is used to summarize the call information during the call.

[0111] S201: Generate summary card

[0112] The answering interface is displayed, and the AI ​​assistant application obtains the audio signal of the caller in real time, identifies the preset type of information in the audio signal to obtain the key content text, and generates a summary text based on the audio signal. The call application generates an information card based on the key content text, and generates a summary card based on the summary text, which is displayed on the answering interface.

[0113] After the call is connected (whether in AI answering mode or manual answering mode), the call application enters the answering interface. After entering the answering interface, the AI ​​assistant application obtains the audio signal received by the call application in real time, and analyzes the audio signal to identify the preset type of information (key content text) and obtain summary information (summary text) within the preset time. For example, after the audio signal is converted from speech to text, the obtained text is input into the large language model to obtain the key content text and summary text.

[0114] In this solution, the preset types of information are: phone numbers, addresses, schedules, etc. When the preset types of information are identified in the audio, the AI ​​assistant application extracts the preset types of information separately to obtain the key content text. Specifically, the large language model can be trained to have the ability to recognize the phone number format of each country and the ability to recognize the address description, so that when the text is input into the large language model, the text content that conforms to the phone number format / address description is extracted separately.

[0115] In one implementation, the AI ​​assistant usually also has the ability to summarize and generalize, and can summarize the caller's speech to obtain a summary text. The summary text can be generated in units of time, for example, a summary text is generated every preset time (such as 1 minute). It is also possible to segment the audio sentences based on the audio pause time of the caller in the audio, with a pause time exceeding a preset time (such as 5s), and generate a summary text according to the audio sentences. Of course, the AI ​​assistant can also segment the sentences by itself and generate a summary text in units of sentences. Alternatively, the AI ​​assistant makes topic judgments based on the audio content and generates a summary text by topic.

[0116] In one implementation, the AI ​​assistant application sends the extracted key content text and summary text to the call application. After receiving the key content text and summary text, the call application generates different cards according to different content types and presents them on the call interface. Specifically, an information card corresponding to the key content text is generated, and a summary card corresponding to the summary text is generated, so that the information card and summary card are presented on the answering interface.

[0117] In one implementation, due to the limitation of screen area, the content presented in the answering interface is limited and not able to display all summary cards and information cards. Since information cards usually have less content and are important information, while summary cards are relatively unimportant information, only information cards can be displayed in full and summary cards can be displayed in stacked form. The stacked summary cards only display the latest summary card in full.

[0118] See also Figure 7 , Figure 7 A schematic diagram of generating a summary card provided in an embodiment of the present application. For example, Figure 7 The answering interface is shown in the figure, and an information card A01 "The location is Restaurant B, BB Road, AA District" is displayed above the caller's avatar logo. In the middle of the screen, a summary card B01 "The department will organize a dinner at 19:00 tonight, and the location is Restaurant B, BB Road, AA District" is displayed above the AI ​​assistant icon.

[0119] S202: Generate summary card and information card

[0120] When a preset operation is detected, the call card interface is entered, and all information cards and summary cards of the call process are displayed on the call card interface. If the user expects to view all summary cards, the information cards and summary cards of the entire call process can be displayed through the preset operation. For example, swipe right to enter the call card interface, and all information cards and summary cards are displayed on the call card interface.

[0121] See also Figure 8 , Figure 8 A schematic diagram of entering a call card interface from an answering interface is provided in an embodiment of the present application. For example, Figure 8 A schematic diagram of entering the call card interface from the answering interface is shown. Slide a preset distance to the right on the left side of the answering interface screen (or any position) to enter the call card interface. The call card interface displays all information cards and summary cards, among which key information corresponds to information cards, and call summaries correspond to summary cards.

[0122] After the information card and summary card are generated, they will exist even if the call is hung up, until the call application is exited. After exiting the call application, the information card / summary card is destroyed. Of course, a folder can also be assigned to each call to store the information card and summary card in the folder.

[0123] Information cards and summary cards can also support interaction. For example, long press to add an information card / summary card to a note / memo / contact. Specifically: after long pressing the card, a secondary menu appears at the corresponding card position. According to the selected secondary menu option, the content of the information card / summary card is added as a new note / contact, etc.

[0124] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0125] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0126] The electronic device in the embodiment of the present application is described in detail below:

[0127] See also Fig. 9 , Fig. 9 A schematic diagram of an electronic device of an AI assistant provided in an embodiment of the present application includes an input module 101, a detection module 102, a text generation module 103 and a display module 104.

[0128] The detection module 101 is used to detect whether the AI ​​call mode is enabled, and the AI ​​call mode refers to a mode in which an AI assistant is used to talk to a callee. The detection module is also used to detect a first operation, and the first operation is used to indicate that the AI ​​call mode is switched to a manual call mode, and the manual call mode refers to a mode in which a user talks to a callee.

[0129] The input module 102 is used to input a prompt word, generate a call voice signal based on the prompt word, and send the call voice signal to the caller.

[0130] The text generation module 103 is used to generate prompt words. The text generation module 103 generates text information according to the received audio signal, identifies the preset type of audio data and generates summary text and summary text based on the large prediction model. The text generation module generates information cards based on the summary text. The text generation module generates summary cards based on the summary text.

[0131] The display module 104 is used to display information cards and summary cards, wherein the information cards are used to indicate the key content text in the call information, and the summary cards are used to indicate the summary content text in the call information. The display module is also used to display the AI ​​assistant icon, which is used to interact with the user to switch the mode.

[0132] In one possible implementation, the detection module 102 detects that the AI ​​call mode is enabled, and the AI ​​assistant conducts a voice call with the caller. During the call, the input module 101 allows the user to input text data, voice data, and select the content of the control according to actual needs and the call content. The detection module 102 detects the input data in real time, and adjusts the call content and call status of the AI ​​assistant according to the input data, so that the user can interact with the AI ​​assistant in real time during the call between the AI ​​and the caller. In addition, the display interface will display the AI ​​assistant icon and prompt words in real time. The prompt words can be dynamically generated based on the audio data, or can be input by the user in real time, or can be of a fixed type. The above prompt words can control the call process and call time by controlling the call reply content and tone of the AI ​​assistant.

[0133] In one possible implementation, the electronic device supports switching of call modes through an AI assistant icon, for example, switching from AI call mode to manual call mode; or switching from manual call mode to AI call mode.

[0134] The above-mentioned electronic devices are generally devices with AI assistants, which support AI assistants to answer calls and allow users to interact with AI assistants in AI call mode. The skills ensure that the AI ​​assistants make calls according to the user's wishes and can effectively ensure the quality of the calls. In addition, the electronic device can also generate corresponding summary text and summary text based on the audio data of the call. It is convenient for users to view and record the content of the meeting.

[0135] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0136] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that contains one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state hard disk (SSD)), etc.

Claims

1. A call interface interaction method, characterized in that: Applied to electronic devices with AI assistant applications, including: Detecting that the AI ​​call mode is enabled, where the AI ​​call mode refers to a mode in which the AI ​​assistant is used to talk to a caller; A first operation is detected, where the first operation is used to instruct to switch the AI ​​call mode to the manual call mode, where the manual call mode refers to a mode in which the user has a call with the callee.

2. The method according to claim 1, characterized in that The method further includes: detecting input of a prompt word, generating a call voice signal based on the prompt word, and sending the call voice signal to the caller.

3. The method according to claim 1, characterized in that The method further comprises: The call interface displays information cards and summary cards, wherein the information card is used to indicate key content text in the call information, and the summary card is used to indicate summary content text in the call information.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The call interface displays an AI assistant icon, which is used to interact with the user to switch modes.

5. The method according to claim 2, characterized in that: The prompt words specifically include fixed recommendation words, and the fixed recommendation words include tone recommendation words and control recommendation words; The tone recommendation words are used to indicate the tone of the output voice text; The control-type recommended words are used to output voice text to control the progress and duration of the call.

6. The method according to claim 2, characterized in that The prompt words specifically include real-time recommendation words, which are text data generated according to real-time call content and are used to reply to the caller.

7. The method according to claim 2, characterized in that The prompt word specifically includes user input text, which is manually input by the user and is text data used to express the user's wishes and ideas.

8. The method according to claim 2, characterized in that: The prompt word specifically includes a user input voice, and the method includes: A second operation is detected, where the second operation is used to indicate receiving the user input voice; Convert the user input voice into AI assistant voice; And send the AI ​​assistant voice to the caller.

9. The method according to claim 1, characterized in that: The method further comprises: A third operation is detected, and the conversation speed of the AI ​​assistant is adjusted.

10. The method according to claim 3, characterized in that: The call interface displays an information card and a summary card, wherein the information card is used to indicate key content text in the call information, and the summary card is used to indicate summary content text in the call information.

11. The method according to claim 10, characterized in that The call interface displays information cards and summary cards specifically including: An audio signal is detected, and information of a preset type in the audio signal is identified to obtain a key content text and a summary text; Generate an information card based on the key content text; A summary card is generated based on the summary text.

12. The method according to claim 10, characterized in that The preset types in the audio signal include but are not limited to telephone numbers, addresses, and schedules.

13. The method according to claim 10, characterized in that The summary text is a text content that summarizes the call content within a preset time period.

14. An electronic device carrying an AI assistant, characterized in that: The device comprises: A detection module, used to detect the activation of an AI call mode, where the AI ​​call mode refers to a mode in which the AI ​​assistant is used to talk to a caller; The detection module is further used to detect a first operation, where the first operation is used to instruct to switch the AI ​​call mode to a manual call mode, where the manual call mode refers to a mode in which the user has a call with the callee.

15. The electronic device according to claim 14, characterized in that: The device also includes: The input module is used to input a prompt word, generate a call voice signal based on the prompt word, and send the call voice signal to the caller.

16. The electronic device according to claim 14, characterized in that: The device also includes: The display module is used to display information cards and summary cards, wherein the information cards are used to indicate key content texts in the call information, and the summary cards are used to indicate summary content texts in the call information.

17. The electronic device according to any one of claims 14 to 16, characterized in that: The display module is also used to display an AI assistant icon, and the AI ​​assistant icon is used to interact with the user to achieve mode switching.

18. The electronic device according to claim 15, characterized in that: The prompt words specifically include fixed recommendation words, and the fixed recommendation words include tone recommendation words and control recommendation words; The tone recommendation words are used to indicate the tone of the output voice text; The control-type recommended words are used to output voice text to control the progress and duration of the call.

19. The electronic device according to claim 15, characterized in that: The prompt words specifically include real-time recommendation words, which are text data generated according to real-time call content and are used to reply to the caller.

20. The electronic device according to claim 15, characterized in that: The prompt word specifically includes user input text, which is manually input by the user and is text data used to express the user's wishes and ideas.

21. The electronic device according to claim 15, characterized in that: The prompt word specifically includes a user input voice, and the method includes: A second operation is detected, where the second operation is used to indicate receiving the user input voice; Convert the user input voice into AI assistant voice; And send the AI ​​assistant voice to the caller.

22. The electronic device according to claim 14, characterized in that: The detection module is also used to detect a third operation and adjust the conversation speed of the AI ​​assistant.

23. The electronic device according to claim 14, characterized in that: The display module is used to display information cards and summary cards, wherein the information cards are used to indicate key content text in the call information, and the summary cards are used to indicate summary content text in the call information.

24. The electronic device according to claim 23, characterized in that: The device also includes a text generation module, which is used to generate text information from the detected audio information; specifically including: The audio signal is detected, and information of a preset type in the audio signal is identified to obtain key content text and summary text; The text generation module generates an information card based on the key content text; The text generation module generates a summary card based on the summary text.

25. The electronic device according to claim 23, characterized in that: The preset types in the audio signal include but are not limited to telephone numbers, addresses, and schedules.

26. The electronic device according to claim 23, characterized in that: The summary text is a text content that summarizes the call content within a preset time period.

27. An electronic device carrying an AI assistant, characterized in that: The device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 1 to 13.

28. A computer storage medium, characterized in that The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 13.

29. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 13.

Images