Intelligent interaction method, electronic equipment, computer storage medium and program product

Abstract

The embodiment of the invention discloses an intelligent interaction method, electronic equipment, a computer storage medium and a program product. The electronic equipment can display information of two or more application programs on the screen of the electronic equipment at the same time, and the information of the application programs can be efficiently and dynamically interacted. According to the scheme provided by the embodiment of the invention, barriers among different information displayed on the screen can be broken, screen contents visible to a user are linked, brand-new intelligent interaction experience and result display are provided for the user, and the multi-task processing capability and efficiency in the electronic equipment are improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202411481300.6, and the original application date is October 22, 2024. The entire content of the original application is incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to an intelligent interaction method, an electronic device, a computer storage medium, and a program product. Background Art

[0003] Electronic devices' split-screen features and multi-window display technologies allow users to handle multiple tasks simultaneously, such as opening multiple applications, viewing multiple documents, and watching videos and news simultaneously. Some users even use two or more electronic devices simultaneously to handle multiple tasks simultaneously across multiple screens. Split-screen, multi-window, and multi-screen scenarios can increase the visual dimension and enhance the user experience and operational efficiency.

[0004] For scenarios where users handle multiple tasks simultaneously, an intelligent interaction method needs to be provided to make user operations easier and provide a better user experience. Summary of the Invention

[0005] The embodiments of the present application provide an intelligent interaction method, an electronic device, a computer storage medium, and a program product, which can break down the barriers between different information displayed on the screen, link the screen content visible to the user, provide the user with a new intelligent interaction experience and result display, and improve the multi-tasking capability and efficiency of electronic devices.

[0006] In a first aspect, an intelligent interaction method is provided, which is applied to an electronic device including a screen. The method may include: displaying a first user interface of a first application in a first area of ​​the screen, and displaying a second user interface of a second application in a second area of ​​the screen, where the first application is different from the second application; receiving a first operation, where the first operation includes an operation acting on the second area and does not include an operation acting on the first area; and displaying first information in the second user interface in response to the first operation, where the first information is determined based on information presented in the first user interface.

[0007] When implementing the method of the first aspect, when an electronic device displays information from different applications in separate screen regions, it can analyze the information in the first region and display the results of the analysis in the second region. This allows the electronic device to simultaneously display information from two applications, breaking down barriers and enabling dynamic information interaction. This method can break down barriers between different information displayed on the screen, allowing the user's visible screen content to be linked, providing the user with a new intelligent interactive experience and display of results, and improving the multitasking capabilities and efficiency of electronic devices.

[0008] In conjunction with the first aspect, in some embodiments, the electronic device may further refresh the content displayed in the first area in response to the first operation, or in response to a second operation performed by the user on the second user interface after the first operation. The second operation is different from the first operation.

[0009] The electronic device refreshes the content displayed in the first area in the following two ways:

[0010] 1. The electronic device refreshes the information displayed by the first application in the first area.

[0011] Mode 1 is equivalent to the electronic device still displaying the information of the first application in the first area. Specifically, the electronic device may refresh the information of the first application in the first area in the following ways:

[0012] (1) The electronic device scrolls to display the first basis in the first user interface, or highlights the first basis in the first user interface.

[0013] The electronic device determines the first information based on a first basis in the information presented by the first user interface, that is, the first basis is to determine the key content of the first information. The first basis can be in the form of text, chart, image, video, link, card, chat record, etc.

[0014] Through the method (1), the electronic device can scroll or highlight the basis of the answer on the right side in the first user interface, which allows the user to easily view the basis and also indicates that the first information provided by the second application has a basis to follow, which can enhance the user's trust in the second application and provide the user with a better interactive experience with the second application.

[0015] (2) The electronic device adds second information in the first user interface, and the second information is determined based on the information presented in the first user interface.

[0016] The second method allows the electronic device to add some new information to the first user interface, thereby providing more information to the user. In addition, this method allows the information of the two applications to interact further by backfilling the content to the first user interface.

[0017] (3) The electronic device displays the fourth user interface of the first application in the first area.

[0018] Method (3) is equivalent to the electronic device switching the interface of the first application to the fourth user interface in the first area. The fourth user interface and the first user interface have different functions. The function of the fourth user interface may be related to the first operation. Method (3) allows the electronic device to switch to the user interface of other functions within the same APP, thereby providing users with more functions.

[0019] 2. The electronic device displays a third user interface of a third application in the first area, where the third application is different from the first application.

[0020] Mode 2 is equivalent to the electronic device jumping to display a user interface of another application in the first area, thereby providing the user with the functions of another application.

[0021] The above paragraphs provide several methods for refreshing the information of the first application in the first area. In the embodiment of the present application, the electronic device can perform a corresponding refresh action in the first area of ​​the electronic device based on the actual needs of the user (such as the first operation or the second operation). When the user's needs are different, the electronic device can perform different refresh actions.

[0022] In conjunction with the first aspect, in some embodiments, the method may further include: displaying third information in the second user interface, where the third information indicates the basis for the first information in the first user interface. With this embodiment, the electronic device can indicate the basis in the second user interface, indicating that the first information provided by the second application has a valid basis. This can enhance the user's trust in the second application and provide a better interaction experience between the user and the second application.

[0023] In conjunction with the first aspect, in some embodiments, the first operation includes inputting a first question. Before displaying the first information in the second user interface, the method may further include: determining that the first question contains a preset designated word; or determining that the first question is relevant to the information presented in the first user interface. Through this embodiment, the electronic device can display information in the second area of ​​the screen in combination with the information in the first area in specific circumstances, thereby meeting the actual needs of users and avoiding abuse of the method provided by this application.

[0024] In combination with the previous embodiment, in some embodiments, the first question input is in one or a combination of the following modes: voice, text, image, video, link, card, chat history, document.

[0025] In conjunction with the previous embodiment, in some embodiments, the modality of the first question input includes a voice modality, and the method may further include: in response to the first operation, displaying a second text in the second user interface, where the second text is a text converted from the first question in the voice modality. This facilitates the user to view the question they have input in the second user interface.

[0026] In conjunction with the first aspect, in some embodiments, before receiving the first operation, the method may further include: displaying one or more question options in the second user interface, where each question option corresponds to a question, and the questions corresponding to the one or more question options are determined based on the information presented in the first user interface; and the first operation includes an operation acting on the first question option in the one or more question options. This can predict the user's intention in advance, simplify the user's subsequent operations, and eliminate the need for the user to enter additional questions in the second application.

[0027] In combination with the first aspect, in some implementations, the electronic device has one screen, and the electronic device can specifically display a first user interface of a first application and a second user interface of a second application in split-screen display.

[0028] In combination with the previous embodiment, the electronic device may display the first user interface of the first application and the second user interface of the second application in split screen mode in the following situations:

[0029] 1. Before the electronic device displays a first user interface of a first application in a first area of ​​the screen and displays a second user interface of a second application in a second area of ​​the screen, it receives a user input to display the first and second applications in a split-screen mode. This is equivalent to the user inputting the split-screen operation, thereby triggering the electronic device to split the screen.

[0030] 2. The electronic device first displays a second user interface on the screen and receives a second question input in the second user interface, where the second question is related to the first application; then, in response to the input of the second question, the electronic device opens the first user interface of the first application in the first area of ​​the screen and displays the second user interface of the second application in the second area of ​​the screen.

[0031] In the second case, it is equivalent to the second application triggering the electronic device to launch the first application in response to the input of the second question. The second question is related to the first application, which means that the first application can solve the second question. For example, when the second question involves navigation, the first application is a map application, or when the second question involves translation, the first application is a translation application. The intelligent assistant can analyze the second question and the various applications installed in the electronic device to determine the first application and trigger its launch.

[0032] 3. The electronic device first displays the second user interface on the screen, receives the first document and the third question input in the second user interface, and the third question is related to the first document; then, in response to the input of the first document and the third question, opens the first user interface of the first application in the first area of ​​the screen, and displays the second user interface of the second application in the second area of ​​the screen, wherein the first user interface is the interface for displaying the first document in the first application. There is no limit to the order in which the first document and the third question are input, both can be input at the same time, or the first document can be input first, or the third question can be input first. In the above three cases, the split-screen style of the electronic device can be left-right split-screen, top-bottom split-screen, floating window split-screen, etc., which are not limited here.

[0033] In conjunction with the first aspect, in some embodiments, the electronic device has one screen, and the first area and the second area are two areas of the electronic device that display separate windows. Specifically, the first area is the display area of ​​the first window on the screen, and the second area is the display area of ​​the second window on the screen.

[0034] In conjunction with the first aspect, in some embodiments, an electronic device has multiple screens. For example, when the electronic device includes a first screen and a second screen, the first area is the display area of ​​the first screen, and the second area is the display area of ​​the second area. In this way, the electronic device can present information from different applications on different screens, and the information presented on different screens can also interact, providing better services to users.

[0035] In conjunction with the first aspect, in some embodiments, an electronic device includes: a first device having a first screen; a second device having a second screen; the first area being the display area of ​​the first screen; and the second area being the display area of ​​the second screen. In this manner, multiple electronic devices can display information from different applications, and the second device can analyze the information displayed by the first device, with the results of the analysis displayed on the second device. In this manner, the simultaneous display of application information by two electronic devices can break down barriers and enable dynamic information interaction.

[0036] In combination with the first aspect, in some embodiments, before displaying the first information in the second user interface, the method may also include: taking a screenshot of the first user interface to obtain a first screenshot, and obtaining the information presented in the first user interface from the first screenshot; or, obtaining the information presented in the first user interface from the operating system of the electronic device.

[0037] In conjunction with the first aspect, in some embodiments, the second application is an intelligent assistant application. Thus, after the user inputs a question to the intelligent assistant through the first operation, the intelligent assistant can answer the user based on the information displayed in the first area of ​​the screen, providing the user with a new intelligent interactive experience and result display, thereby improving the multitasking capability and efficiency of the electronic device.

[0038] In a second aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of the first aspect or any one of the embodiments of the first aspect.

[0039] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method of the first aspect or any one of the embodiments of the first aspect is implemented.

[0040] In a fourth aspect, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, it implements the method of the first aspect or any one of the embodiments of the first aspect.

[0041] In a fifth aspect, a chip system is provided, which includes a processing circuit and an interface circuit. The interface circuit is used to receive computer instructions and transmit them to the processing circuit. The processing circuit is used to run the computer instructions to implement a method as described in the first aspect or any one of the embodiments of the first aspect.

[0042] In a sixth aspect, an intelligent interaction method is provided, applied to a communication system, the communication system comprising: a first device including a first screen, a second device including a second screen, the first device and the second device establishing a communication connection. The method may include: the first device displaying a first user interface of a first application on the first screen; the second device displaying a second user interface of a second application on the second screen, the second application being different from the first application; receiving a first operation, the first operation including an operation on the second screen and not including an operation on the first screen; and the second device displaying first information on the second user interface in response to the first operation, the first information being determined based on information presented in the first user interface.

[0043] In implementing the method of the sixth aspect, when two electronic devices display information from different applications on their screens, one electronic device can analyze the information presented on the screen of the other electronic device and display the results of the analysis on its own screen. In this way, the information of the two applications displayed by the two electronic devices can break down barriers and achieve dynamic interaction of information. This method can break down the barriers between different information displayed on the screens of different devices, allowing the screen content visible to the user to be linked, providing the user with a new intelligent interactive experience and result display, and improving the ability and efficiency of multiple electronic devices to handle multiple tasks.

[0044] In conjunction with the sixth aspect, in some embodiments, the second device may further refresh the content displayed in the first area in response to the first operation, or in response to a second operation performed by the user on the second user interface after the first operation. The second operation is different from the first operation. The manner in which the second device refreshes the content displayed in the first area can refer to the manner in which the electronic device refreshes the content displayed in the first area in the first aspect, and is not further described here.

[0045] In conjunction with the sixth aspect, in some embodiments, the method may further include: the second device displaying third information in the second user interface, where the third information indicates the basis for the first information in the first user interface. With this embodiment, the second device can indicate the basis in the second user interface, indicating that the first information provided by the second application has a valid basis. This can enhance the user's trust in the second application and provide a better interactive experience between the user and the second application.

[0046] In conjunction with the sixth aspect, in some embodiments, the first operation includes inputting a first question, and before the second device displays the first information in the second user interface, the method may further include: the second device determining that the first question contains a preset designated word; or the second device determining that the first question is relevant to the information presented in the first user interface. Through this embodiment, the second device can display information in the second area of ​​the screen in combination with the information in the first area in specific circumstances, thereby meeting the actual needs of users and avoiding abuse of the method provided in this application.

[0047] In conjunction with the sixth aspect, in some embodiments, before the second device receives the first operation, the method may further include: the second device displays one or more question options in the second user interface, where each question option corresponds to a question, and the questions corresponding to the one or more question options are determined based on information presented in the first user interface; and the first operation includes an operation acting on the first question option in the one or more question options. This can predict the user's intention in advance, simplify the user's subsequent operations, and eliminate the need for the user to enter a question into the second application on the second device.

[0048] In combination with the sixth aspect, in some embodiments, the second application is an intelligent assistant application.

[0049] In combination with the sixth aspect, in some embodiments, before displaying the first information in the second user interface, the method may further include: the second electronic device obtains the information presented in the first user interface from the first device through a communication connection.

[0050] In conjunction with the previous embodiment, the first device can capture the first user interface to obtain a first screenshot, and obtain information presented in the first user interface from the first screenshot; alternatively, the first device can obtain the information presented in the first user interface from the operating system. The first device can then send the obtained information to the second device.

[0051] In the seventh aspect, a communication system is provided, which includes a first device and a second device, wherein the first device is used to perform the operations performed by the first device in the method of the sixth aspect or any one of the embodiments of the sixth aspect, and the second device is used to perform the operations performed by the second device in the method of the sixth aspect or any one of the embodiments of the sixth aspect.

[0052] In the eighth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the operations performed by the first device or the second device in the method of the sixth aspect or any one of the embodiments of the sixth aspect.

[0053] In the ninth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the operation performed by the first device or the second device in the method of the sixth aspect or any one of the embodiments of the sixth aspect is implemented.

[0054] In the tenth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, it implements the operations performed by the first device or the second device in the method of the sixth aspect or any one of the embodiments of the sixth aspect.

[0055] In the eleventh aspect, a chip system is provided, which includes a processing circuit and an interface circuit, the interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to run the computer instructions to implement the operations performed by the first device or the second device in the method of the sixth aspect or any one of the embodiments of the sixth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 An example of a user interface that provides an intelligent question-and-answer function on an electronic device with split screen is shown;

[0057] Figure 2 Another set of user interfaces for providing intelligent question-and-answer functions on an electronic device is exemplarily shown;

[0058] Figure 3 An example of a group of user interfaces that provide navigation functions on an electronic device with split screens is shown;

[0059] Figure 4 An example of a user interface showing a split screen electronic device providing an intelligent question-and-answer function and a navigation function;

[0060] Figure 5 An example of a group of user interfaces that provide document analysis functions on an electronic device with split screens is shown;

[0061] Figure 6An example of a group of user interfaces that provide a form processing function on an electronic device with split screen is shown;

[0062] Figure 7 An example of a group of user interfaces providing a translation function on an electronic device with split screen is shown;

[0063] Figure 8 An example of a set of user interfaces that provide intelligent question-and-answer functions in multiple windows on a single electronic device is shown;

[0064] Figure 9 An example of a user interface in which two electronic devices jointly provide an intelligent question-answering function is shown;

[0065] Figure 10 A flowchart of the intelligent interaction method provided in an embodiment of the present application;

[0066] Figure 11 A flowchart of another intelligent interaction method provided in an embodiment of the present application;

[0067] Figure 12 A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0068] Figure 13 A software structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the accompanying drawings.

[0070] The embodiments of the present application provide an intelligent interaction method, which can be applied to electronic devices such as non-folding screen mobile phones with one screen, folding screen mobile phones (such as two-folding mobile phones, three-folding mobile phones or more folding mobile phones, etc.), tablet computers, car computers, laptops, large-screen TVs, desktop computers, desktop computers with touch-sensitive surfaces or touch panels, and can also be applied to two or more electronic devices used in conjunction. A folding screen mobile phone can have a whole foldable flexible display screen, or it can have multiple screens that can be combined into one display screen. When the folding screen mobile phone executes this method, it is in a fully unfolded state or a partially unfolded state (such as a three-folding mobile phone with one screen folded and two screens in an unfolded state) or a folded state.

[0071] In an embodiment of the present application, an electronic device can simultaneously display information from different applications (APPs) in a split-screen or multi-window scenario. Multiple electronic devices used in conjunction provide multiple screens and can also simultaneously display information from different applications (APPs).

[0072] In one embodiment of the present application, an electronic device can simultaneously display information of two or more applications on its own screen, and the information between these applications can interact dynamically and efficiently. In another embodiment of the present application, multiple electronic devices used in conjunction can respectively display information of different applications on their own screens, and the information between these applications can also interact dynamically and efficiently. Through the method provided in the embodiment of the present application, the barriers between different information displayed on the screen can be broken, and the screen content visible to the user can be linked together, providing the user with a new intelligent interactive experience and result display, and improving the ability and efficiency of multi-tasking in electronic devices.

[0073] The following describes the intelligent interaction method provided by the embodiments of the present application through some specific examples.

[0074] Figure 1-Figure 7 The user interface displayed when the screen is split on a single electronic device is exemplified.

[0075] The single electronic device can be a mobile phone (such as a non-folding screen mobile phone, a folding screen mobile phone, etc.), a tablet computer, a car computer, a laptop computer, a large-screen TV, a desktop computer, etc. The electronic device can have a larger screen size, so that the reading effect during the split-screen display can be guaranteed. Of course, the electronic device can also have a screen of a regular size, and the embodiments of the present application are not limited to this.

[0076] In the embodiment of the present application, split screen refers to the function of dividing a screen into two or more display areas, so that the electronic device can display different content (such as content provided by different applications or different content provided by the same application) on different display areas. The multiple display areas obtained by splitting the screen can be combined to cover the entire screen. When the screen is split up and down or left and right, the multiple display areas obtained by splitting the screen do not overlap with each other; when the screen is split in the style of a floating window, some areas may overlap. The size of the multiple display areas obtained by splitting the screen can be adjusted by the user as needed. The triggering method of splitting the screen can refer to the subsequent Figure 2 Related introduction.

[0077] Figure 1 A set of user interfaces for providing intelligent question-and-answer functions on an electronic device with split screens is exemplified.

[0078] refer to Figure 1 (a), the electronic device displays the user interface 10 after starting the smart assistant.

[0079] An intelligent assistant is a feature provided by electronic devices such as mobile phones, tablets, and vehicles. It enables the intelligent device to detect user input questions (such as detecting user input voice through a microphone, detecting user input text or documents through a display, etc.), recognize the user's intention, and respond in a targeted manner, such as answering voice, launching applications, changing device configurations, etc. The intelligent assistant in the embodiments of the present application enables electronic devices to link information displayed on the screen from different applications, providing users with a new intelligent interactive experience and result display.

[0080] The smart assistant can be a voice assistant, such as the voice assistant "Xiao Yi" provided by Huawei, and voice assistants provided by other manufacturers. The smart assistant can also be a chat robot model based on artificial intelligence. The smart assistant can be an APP in the electronic device (it can be a system APP or a third-party APP), or it can be a function started through a web page, or it can be a small program. The electronic device can detect the user's voice input. The electronic device using speech recognition technology, natural language processing (such as semantic understanding) technology can start the voice assistant after detecting the wake-up word (such as "Xiao Yi Xiao Yi"), or it can start the voice assistant after detecting the user operation acting on the electronic device (such as long pressing the power button, pressing the voice button on the vehicle). In some other embodiments, the electronic device can also always turn on the voice assistant, which is not limited here. The smart assistant is just a name used in the embodiment of the present application. In other embodiments, the smart assistant can also be called other names, such as smart question and answer application, question and answer robot, etc., which are not limited in the embodiment of the present application.

[0081] like Figure 1 As shown in (a), the user interface 10 displays a dialog box 101 between the smart assistant and the user. Dialog box 101 displays the smart assistant name 101a, the user's avatar 101b, text converted from the user's voice input 101c, a keyboard icon 101d, a voice input icon 101e, and an add control 101f. Dialog box 101 can be displayed in the center of the screen.

[0082] The name of the smart assistant 101a is, for example, "Xiaoyi", which is located on the left side of the dialog box 101, and the user avatar 101b is displayed on the right side of the dialog box 101, which means that the left side of the dialog box 101 displays the answer of the smart assistant, and the right side displays the question entered by the user.

[0083] After the electronic device starts the smart assistant, it can collect the user's input voice through the microphone, and convert the user's voice into text through voice recognition technology and display it in the dialog box 101, such as Figure 1The text 101c in (a) of FIG. 101 is displayed. The keyboard icon 101d can be used to receive user operations. In response to the user operations, the electronic device can stop collecting voice through the microphone and display a virtual keyboard in the dialog box 101 for the user to enter text. The voice input icon 101e changes dynamically after the microphone of the electronic device is started (such as a flowing dynamic effect inside) to prompt the user that the current electronic device is collecting their voice. Adding controls 101f supports users to input more information into the smart assistant, such as pictures, documents, etc.

[0084] like Figure 1 As shown in (a), the user inputs a voice to the intelligent assistant, and the text 101c converted from the voice "Huawei and Avita Investment Agreement, how much share?" is displayed in the dialog box 101.

[0085] Afterwards, the smart assistant understands and analyzes the questions input by the user through technologies such as natural language processing (such as semantic understanding), and then determines whether it is necessary to call other APPs when answering the questions input by the user. For example, if the user just wants the smart assistant to perform simple operations such as adding a schedule and checking the weather, the smart assistant itself can provide the corresponding services without calling other APPs; if the user needs the smart assistant to query some complex information, navigation, etc., the smart assistant itself may not be able to provide the corresponding services, and thus needs to call other APPs. The APP called by the smart assistant can be the default setting of the electronic device, or it can be set by the user, or it can be matched by the smart assistant according to the question input by the user. For example, if the user's question involves food recommendations, a food recommendation APP is called, and if the user's question involves location inquiries and navigation, a map APP is called. For example, if Figure 1 As shown in (a), the question entered by the user includes query data. The smart assistant itself does not contain such information and needs to call other apps (such as browsers) to query this data.

[0086] like Figure 1As shown in the user interface 20 shown in (b), after the intelligent assistant analyzes the text 101c, it triggers the electronic device to start the browser, and then the intelligent assistant sends an instruction to the browser according to the text 101c, which is used to simulate the operation (such as touch operation, voice instruction) input by the user to the browser, thereby triggering the browser to search for the problem represented by the text 101c. Alternatively, the intelligent assistant can also call the application program interface (API) supported by the browser to trigger the browser to search for the problem represented by the text 101c. After that, the electronic device can search for relevant content (such as encyclopedia, news, etc.) related to the text 101c through the browser, and can display the relevant content provided by the browser on the left side of the screen, and switch the dialog box 101 to the right side of the screen for display. If the browser searches for relevant content of multiple web pages, the relevant content of the multiple web pages can also be returned to the intelligent assistant, and the intelligent assistant will filter out a web page from the multiple web pages, and then trigger the browser to display the content of the web page.

[0087] like Figure 1 As shown in (b), the smart assistant triggers the electronic device to split the screen into left and right areas, dividing the screen into a left area 102a and a right area 102b. The areas of the left area 102a and the right area 102b are the same by default. There is a separator bar 103 where the left area 102a and the right area 102b meet. The separator bar 103 can be used to receive user operations (such as dragging to the left and dragging to the right), and the electronic device can adjust the areas of the left area 102a and the right area 102b in response to the user operation. The separator bar 103 can also be used to receive user operations to close the split screen.

[0088] The left area 102a displays a page 104 provided by the browser, and the page 104 displays news related to the text 101c that is searched by the browser.

[0089] The right area 102b displays a dialog box 101, in which the intelligent assistant's answer to the user's input question, i.e., text 101g, is added. The text 101g can be the answer to the user's input question synthesized by the intelligent assistant through Internet search, local knowledge base search, etc., through content integration and splicing, and combined with the intelligent assistant's own knowledge. For example, Figure 1 In (b), text 101g reads "According to the news on August 20, Avita holds a 10% stake." Furthermore, the intelligent assistant finds the relevant portion of text 101g, namely, evidence 104a, within the content displayed in left area 102a, and triggers the browser to scroll to evidence 104a, or triggers the electronic device to highlight evidence 104a. This highlighting can be done by highlighting, adding a border, making the font bolder, changing the font color, and so on.

[0090] The smart assistant may obtain the content displayed in the left area 102a in the following two ways:

[0091] 1. The smart assistant obtains this content from the bottom layer through the operating system (OS). The browser runs in the OS, and all content displayed by the browser is displayed on the screen through the OS. Therefore, the OS can obtain the content displayed by the browser, and the smart assistant can obtain this content through the OS. If the height of the left area 102a is limited, only part of the content of the page is displayed in the left area 102a, and there is other content below the displayed content that can be turned to display, the smart assistant can obtain the content of the complete page through the OS, including the content not currently displayed in the left area 102a.

[0092] 2. The smart assistant can trigger the electronic device to take a full-screen screenshot, or take a screenshot of the left area 102a, and then extract the content from the screenshot obtained.

[0093] exist Figure 1 In the illustrated user interface, the application corresponding to the left area 102a and the application corresponding to the right area 102b interact with each other in at least three ways: the intelligent assistant transmits text 101g displayed in the right area 102b to the browser, which searches for relevant content and displays page 104 in the left area 102a; the intelligent assistant retrieves the browser's content and responds to the user based on it, i.e., displays text 101g; and the intelligent assistant triggers the left area 102a to scroll to or highlight a relevant portion of text 101g, i.e., based on 104a. Thus, from the user's perspective, the information displayed in the left area 102a and the right area 102b is no longer isolated, but rather interacts with each other, forming a complete context. This not only displays more information on the screen, but also reorganizes the information, improving the intelligent assistant's comprehension ability and enhancing the user experience.

[0094] exist Figure 1 In the illustrated user interface, the right area 102b displays the conversation between the intelligent assistant and the user, while the left area 102a highlights the basis for the intelligent assistant's answer. Thus, when the user interacts with the intelligent assistant, they can see their questions and answers in the right area 102b and the basis for the intelligent assistant's answer from the content displayed in the left area 102a. This demonstrates that the intelligent assistant's answers are well-founded, which can enhance the user's trust in the intelligent assistant and provide a better interactive experience for the user and the intelligent assistant.

[0095] Figure 2 Another set of user interfaces for providing intelligent question-and-answer functions on an electronic device with split screen is exemplified.

[0096] refer to Figure 2 (a), the electronic device starts the browser and the smart assistant, and displays the two applications in split screen in the user interface 30. The user interface 30 can be a split screen interface started by the user in any of the following ways: finger joint operation, long press the recent key in the system navigation bar, two-finger swipe up operation from the bottom of the screen, three-finger swipe up operation inside the screen, starting split screen on the multitasking page, starting split screen through the side application bar, switching split screen with a floating window, etc. The user interface 30 is triggered by the user's split screen operation, and Figure 1 The user interface 20 shown in (b) is the interface displayed after the smart assistant analyzes the user's intention and triggers the electronic device to split the screen.

[0097] The user interface 30 displays: a page 104 provided by the browser in the left area 102a of the screen, a dialog box 101 between the smart assistant and the user in the right area 102b of the screen, and a separator 103. The contents of the dialog box 101 can be found in Figure 1 Related introduction.

[0098] refer to Figure 2 (b) After the electronic device splits the screen, the user inputs a voice message to the intelligent assistant, and the text 101h converted from the voice message "Whose strategic intention does the investment in the display message reflect?" is displayed in the dialog box 101.

[0099] Afterwards, the intelligent assistant uses natural language processing (e.g., semantic understanding) and other technologies to understand and analyze the text 101h and determine whether it is necessary to combine information displayed in other areas of the screen (e.g., information displayed in the left area 102a of the screen) to answer the user's question. The intelligent assistant can determine whether it is necessary to combine information displayed in other areas of the screen to answer the user's question in the following two ways:

[0100] 1. Text 101h contains designated words. These designated words may be preset by the electronic device or by the user, such as "left screen," "right screen," "other content on the screen," "display message," "left picture," etc. These designated words indicate that the user's question is related to other information displayed on the screen.

[0101] 2. The text 101h is related to the information displayed in other areas of the screen. This can be determined by the intelligent assistant after analyzing the text 101h and the information displayed in other areas of the screen. For example, the intelligent assistant can determine whether the two have the same keywords. If so, it is determined that the two are related. Figure 2 In (b), the left area 102a and the right area 102b both contain the keyword "investment", so it can be determined that the text 101h is related to the information displayed in other areas of the screen.

[0102] For more information about how the smart assistant can obtain information displayed in other areas of the screen, please refer to Figure 1 For example, you can obtain it through screenshots or through the OS.

[0103] refer to Figure 2 (c), if the smart assistant determines that it needs to combine the information displayed in other areas of the screen (such as the information displayed in the left area 102a of the screen) to answer the user's question, the electronic device adds the smart assistant's answer to text 101h in the dialog box 101, that is, text 101i. Text 101i is the answer to text 101h found by the smart assistant from the content displayed in the left area 102a after the smart assistant obtains the content. For example, Figure 2 In (c), text 101i reads "The news on the left shows that the investment reflects Changan Automobile's strategic intention." Furthermore, the intelligent assistant finds the relevant portion of text 101i, namely, evidence 104b, within the content displayed in left area 102a and triggers the browser to scroll to evidence 104b, or triggers the electronic device to highlight evidence 104b. This highlighting can be done by highlighting, adding a border, making the font bolder, changing the font color, and so on.

[0104] If the intelligent assistant determines that it does not need to combine information displayed in other areas of the screen to answer the user's question, the intelligent assistant can use its own knowledge to answer the user's question.

[0105] exist Figure 2 In the illustrated user interface, the application corresponding to left area 102a and the application corresponding to right area 102b interact with each other in at least two ways: the intelligent assistant retrieves the browser content and responds to the user based on it, i.e., displays text 101i; and the intelligent assistant triggers left area 102a to scroll to or highlight a related portion of text 101i, i.e., based on 104b. Thus, from the user's perspective, the information displayed in left area 102a and right area 102b is no longer isolated, but rather interacts with each other, forming a complete context. This not only utilizes the screen to display more information, but also achieves information reorganization, improving the intelligent assistant's comprehension ability and enhancing the user experience.

[0106] exist Figure 2 In the illustrated user interface, the right area 102b displays the conversation between the intelligent assistant and the user, while the left area 102a highlights the basis for the intelligent assistant's response. This allows the user to perceive that the intelligent assistant incorporates information from other areas of the screen when responding, satisfying the user's intent and enhancing the user experience.

[0107] Figure 3 A group of user interfaces in which an electronic device provides a navigation function in split screen is exemplified.

[0108] refer to Figure 3 (a), the electronic device starts the smart assistant and displays the user interface 10. The user interface 10 can refer to Figure 1 (a). Figure 3 As shown in (a), the user interface 10 displays a text 101j "What are the driving routes to Restaurant 2?" converted from the voice input by the user.

[0109] After the smart assistant analyzes text 101j and finds that the text 101j involves route inquiries and navigation, it triggers the electronic device to launch a map APP. The smart assistant then sends text 101j to the map APP, so that the map APP can analyze the text 101j, automatically set the location of the keyword "Restaurant 2" in the text 101j as the destination, and query the route from the current location to the destination, and then display the query results on the screen. In other embodiments, the smart assistant can convert text 101j into an instruction that can be recognized by the map APP, and then send the instruction to the map APP. The instruction is used to simulate the user's input operation on the map APP (such as touch operation, voice command), thereby triggering the map APP to query the route from the current location to the destination "Restaurant 2" and display the query results on the screen. Alternatively, the smart assistant can also call the API supported by the map APP to trigger the map APP to query the route from the current location to the destination "Restaurant 2", and then display the query results on the screen.

[0110] like Figure 3 As shown in (b), the smart assistant triggers the electronic device to split the screen into left and right areas, dividing the screen into a left area 102a and a right area 102b. The areas of the left area 102a and the right area 102b are the same by default. There is a separator bar 103 where the left area 102a and the right area 102b meet. The left area 102a displays a page 105 provided by a map APP, and the page 105 displays several routes from the current location to the destination queried by the map APP. The right area 102b displays a dialog box 101, and the dialog box 101 adds the smart assistant's answer to the question input by the user, that is, text 101k. Text 101k is the answer to the question input by the user found from the content displayed in the left area 102a by the smart assistant after the smart assistant obtains the content displayed in the left area 102a, that is, "Okay, 2 driving routes have been planned for you." For how the smart assistant obtains the content displayed in the left area 102a, please refer to Figure 1 Related description.

[0111] refer to Figure 3In (b), the user can continue to input voice into the smart assistant, and the dialog box 101 displays text 101L converted from the user's voice input, "Which places can I refuel on these routes?" The smart assistant analyzes text 101L and finds that text 101L is related to the information displayed in the left area 102a of the screen. It determines that it needs to combine the information in the left area 102a of the screen to answer the user's question, and sends text 101L to the map app. The map app can then analyze text 101L and display the searched gas stations on the multiple routes displayed on page 105. In other embodiments, the smart assistant can also convert text 101L into a command that the map app can recognize, and then send the command to the map app, thereby triggering the map app to query for gas stations and display the searched gas stations on the multiple routes displayed on page 105. Alternatively, the smart assistant can call an API supported by the map app to trigger the map app to query for gas stations and display the searched gas stations on the multiple routes displayed on page 105. Afterwards, the intelligent assistant obtains the content displayed in the left area 102a, finds the answer to the text 101L from the content, and then displays the answer text 101m in the right area 102b of the screen.

[0112] exist Figure 3 In the user interface shown, users can simultaneously view the navigation-related questions they have asked (such as text 101j, text 101L), the response process of the map APP (such as page 105), and the answer results of the smart assistant (such as text 101k, text 101m), which improves the user experience.

[0113] Figure 4 The example shows a set of user interfaces in which an electronic device provides intelligent question-and-answer functions and navigation functions on a split screen.

[0114] refer to Figure 4 (a), the electronic device starts the smart assistant and displays the user interface 10. The user interface 10 can refer to Figure 1 (a). Figure 4 As shown in (a), the user interface 10 displays a text 101n converted from the voice input by the user, "Xiaoyi, help me see what delicious food is nearby."

[0115] After the smart assistant analyzes the text 101n and finds that the text 101n involves food recommendations, it triggers the electronic device to start the food recommendation APP. The smart assistant then sends the text 101n to the food recommendation APP, so that the food recommendation APP can analyze the text 101n, query nearby restaurants, and then display the query results on the screen. In other embodiments, the smart assistant can also convert the text 101n into an instruction that the food recommendation APP can recognize, and then send the instruction to the food recommendation APP, thereby triggering the food recommendation APP to query nearby restaurants and display the query results on the screen. Alternatively, the smart assistant can also call the API supported by the food recommendation APP to trigger the food recommendation APP to query nearby restaurants and display the query results on the screen.

[0116] like Figure 4 As shown in (b), the smart assistant triggers the electronic device to split the screen into left and right areas, dividing the screen into a left area 102a and a right area 102b. The left area 102a displays a page 106 provided by a food recommendation APP, and the page 106 displays the nearby restaurant information queried by the food recommendation APP. The right area 102b displays a dialog box 101, and the dialog box 101 adds the smart assistant's answer to the question input by the user, that is, text 101o. Text 101o is the answer to the question input by the user found from the content displayed in the left area 102a by the smart assistant after the smart assistant obtains the content displayed in the left area 102a, that is, "Help you find these delicacies, please check." For the way in which the smart assistant obtains the content displayed in the left area 102a, please refer to Figure 1 Related description.

[0117] refer to Figure 4 (b), the user can continue to input voice to the smart assistant, and the dialog box 101 displays the text 101p "What special foods does this 'Restaurant 1' have" converted from the user's input voice. The electronic device determines that it needs to combine the information in the left area 102a of the screen to answer the user's question, and sends the text 101p to the food recommendation APP, so that the food recommendation APP can further display the homepage of "Restaurant 1" on page 106. In other embodiments, the smart assistant can also convert the text 101p into an instruction that the food recommendation APP can recognize, and then send the instruction to the food recommendation APP, thereby triggering the food recommendation APP to further display the homepage of "Restaurant 1" on page 106. Alternatively, the smart assistant can also call the API supported by the food recommendation APP to trigger the food recommendation APP to further display the homepage of "Restaurant 1" on page 106.

[0118] Afterwards, if Figure 4As shown in (c), the intelligent assistant obtains the content displayed in the left area 102a, finds the answer to the text 101p from the content, and then displays the answer text 101q in the right area 102b of the screen. At the same time, the intelligent assistant finds the relevant part of the text 101q in the content displayed in the left area 102a, namely the basis 106a, and triggers the browser to scroll to the basis 106a, or triggers the electronic device to highlight the basis 106a. The highlighting method can be highlighting, adding a border, bolding the font, changing the font color, etc.

[0119] like Figure 4 As shown in (c), the user can continue to input voice to the smart assistant, and the dialog box 101 displays the text 101r "Navigate to 'Restaurant 1'" converted from the user's input voice. After analyzing the text 101r, the smart assistant finds that the text 101r involves the information and navigation function of the left area 102a of the screen. The smart assistant then extracts the address of Restaurant 1 from the left area 102a of the screen, and then sends the address to the map APP, or sends the navigation instruction containing the address to the map APP. In addition, the smart assistant triggers the electronic device to start the map APP, so that the map APP can automatically set the address as the destination, query the route from the current location to the destination, and then display the query results on the screen.

[0120] like Figure 4 As shown in (d), the smart assistant triggers the electronic device to launch a map app and displays the map app's query results, i.e., page 107, in the left screen area 102a. A dialog box 101 is displayed in the right screen area 102b. Dialog box 101 includes the smart assistant's response to the user's input, i.e., text 101s: "OK, we've planned the fastest driving route for you." In some embodiments, the map app may also display a navigation page directly in the left screen area 102a, rather than a route display page.

[0121] Figure 5 A set of user interfaces for providing document analysis functions on an electronic device in split screen is exemplified.

[0122] refer to Figure 5 (a), the electronic device starts the smart assistant and displays the user interface 10. The user interface 10 can refer to Figure 1 (a). Figure 5 As shown in (a), the user enters a portable document format (PDF) document in the dialog box 101. The dialog box 101 displays the document logo and the text 101t "Xiaoyi, help me analyze this annual report" converted from the user's input voice.

[0123] After the intelligent assistant analyzes the text 101t and finds that the text 101t involves document analysis, it triggers the electronic device to open the PDF document input by the user and extract the content of the PDF document for analysis, thereby answering the user's question.

[0124] like Figure 5 As shown in (b), the right area 102b of the screen displays a dialog box 101, which displays the intelligent assistant's answer to the user's input question, namely text 101u. Text 101u is the answer given by the intelligent assistant after analyzing the user's input question and the PDF document. Dialog box 101 also displays text 101v, which indicates the basis of the answer given by the intelligent assistant in the PDF document. The left area 102a of the screen displays a PDF document opened by the electronic device, and the document can be scrolled to or highlighted to display the part where the above-mentioned basis is located.

[0125] In some embodiments, after receiving the PDF document input by the user, the intelligent assistant can automatically analyze the form document and automatically propose some question options for the user to choose, such as interpreting the document, reading the full text, translating the full text, extracting text, etc. Figure 5 As shown in (a). This can predict the user's intention in advance and simplify the user's subsequent operations. In this case, after the user selects the question provided by the intelligent assistant, the intelligent assistant defaults to answering the user's question in combination with the information displayed in other areas of the screen (such as the information displayed in the left area 102a of the screen), without the need to perform additional judgments and without the user having to enter additional questions.

[0126] Figure 6 The example shows a group of user interfaces in which an electronic device provides a form processing function in a split screen.

[0127] refer to Figure 6 In (a), the electronic device opens a spreadsheet document in a spreadsheet application, starts an intelligent assistant, and displays the spreadsheet application and the intelligent assistant in a split screen in a user interface 90. The user interface 90 displays: a page 107 provided by the spreadsheet application in the left area 102a of the screen, a dialog box 101 between the intelligent assistant and the user in the right area 102b of the screen, and a separator 103. Page 107 displays a spreadsheet document opened by the electronic device. The contents of the dialog box 101 can be found in Figure 1 Related introduction.

[0128] refer to Figure 6 (b) After the electronic device splits the screen, the user inputs a voice message to the intelligent assistant, and the text 101w converted from the voice message "Xiaoyi, help me draw a line graph of the P value, R value, and F value of all the solutions" is displayed in the dialog box 101.

[0129] The intelligent assistant uses natural language processing (e.g., semantic understanding) and other technologies to understand and analyze text 101w. It discovers that text 101w is related to the information on the left and determines that it needs to combine information displayed in other areas of the screen (e.g., information displayed in the left area 102a of the screen) to answer the user's question. The electronic device then extracts the table content on page 107 in the left area 102a and provides an answer based on the table content, adding a chart 101x to the dialog box 101. Chart 101x is drawn by the intelligent assistant based on the table content in the left area 102a of the screen according to the user's needs.

[0130] refer to Figure 6 (b), the user can continue to input voice to the smart assistant, and the dialog box 101 displays the text 101y converted from the user's voice input, "Help me calculate the average of the P value, R value, and F value of all solutions." The smart assistant analyzes the text 101y and finds that the text 101y is related to the information displayed in the left area 102a of the screen, and determines that it needs to combine the information in the left area 102a of the screen to answer the user's question. Therefore, Figure 6 As shown in (c), the intelligent assistant extracts the table content in the page 107 in the left area 102a, and calculates the numbers in the table according to the user's question, and then presents the calculation results in the dialog box 101, that is, text 101z. In addition, the intelligent assistant can also send the calculation results to the table application, so that the table application can fill the calculation results back into the open table document, as shown in FIG. Figure 6 As shown in (c). The backfilled content can be highlighted. In some embodiments, the content in the left area 102a used by the intelligent assistant to answer the user's question can also be highlighted.

[0131] Figure 7 The present invention exemplarily shows a set of user interfaces in which an electronic device provides a translation function in split screen.

[0132] refer to Figure 7 In (a), the electronic device starts the gallery and the smart assistant, and displays the two applications in a split screen in the user interface 100. The user interface 100 displays: a page 108 provided by the gallery in the left area 102a of the screen, a dialog box 101 between the smart assistant and the user in the right area 102b of the screen, and a separator 103. Page 108 displays a picture opened by the electronic device. The contents of the dialog box 101 can be referred to Figure 1 Related introduction.

[0133] refer to Figure 7(b) After the electronic device splits the screen, the user inputs a voice message to the intelligent assistant, and the text 101A converted from the voice message "Xiaoyi, help me translate the displayed message" is displayed in the dialog box 101.

[0134] The intelligent assistant understands and analyzes text 101W through natural language processing (such as semantic understanding) and other technologies, and finds that text 101W is related to the information on the left, and determines that it is necessary to combine the information displayed in other areas of the screen (such as the information displayed in the left area 102a of the screen) to answer the user's question. Specifically, the intelligent assistant extracts the picture content (such as text) in page 108 in the left area 102a, and then translates the extracted text and displays the translation in the left area 102a. The translation can partially or completely cover the original text in the picture, or it can be displayed in a blank space. In addition, after the intelligent assistant completes the translation, it can also add text 101B in the dialog box 101 "I have translated it for you, and the translation is on the left side of the screen" to prompt the user that the translation is complete. The language of the translation can be the user's common language or a preset language.

[0135] In some implementations, the translation produced by the intelligent assistant may also be displayed in the dialog box 101 , which is not limited here.

[0136] exist Figure 7 In the user interface shown, the user does not need to enter the original text into the smart assistant. The electronic device can translate across screens and display the translated text on one of the screens, making translation easier and faster.

[0137] Figure 8 A set of user interfaces that provide intelligent question-and-answer functions in multiple windows of a single electronic device is exemplified.

[0138] The single electronic device can be a mobile phone (such as a non-folding screen mobile phone, a folding screen mobile phone, etc.), a tablet computer, a car computer, a laptop computer, a large-screen TV, a desktop computer, etc. The electronic device can have a larger screen size, so that the reading effect when multiple windows are displayed can be guaranteed. Of course, the electronic device can also have a screen of a regular size, and the embodiments of the present application are not limited to this.

[0139] refer to Figure 8 , the electronic device starts the browser and the smart assistant, and displays the browser and the smart assistant in two windows respectively. Figure 8As shown, the user interface 110 displays: a browser window 109, and an intelligent assistant window 130. Window 109 and window 130 may partially overlap or not overlap at all. The size of window 109 and the size of window 130 can be adjusted by the user as needed. Window 109 displays the page provided by the browser, and window 130 displays the dialog box between the intelligent assistant and the user. When the user inputs voice to the intelligent assistant, the intelligent assistant can answer the user's question based on the information displayed in window 109, and can also trigger the browser in window 109 to scroll to or highlight the basis for the intelligent assistant's answer. This part can be referred to Figure 1 and Figure 2 Related introduction.

[0140] Multi-window is different from the split screen mentioned above. Multi-window refers to the function of launching multiple windows in one screen, so that the electronic device can display different content in different windows (such as content provided by different applications or different content provided by the same application). The triggering method of multi-window is different from the triggering method of split screen. The electronic device can respond to multiple user operations input successively (such as the operation of launching a certain application) to open the windows of multiple applications in sequence, thus entering a multi-window scenario. Multiple windows can be merged to cover the entire screen or not. Multiple windows can partially overlap or not overlap. The size, position, etc. of each window on the screen can be adjusted individually by the user as needed.

[0141] Figure 9 A set of user interfaces in which two electronic devices jointly provide an intelligent question-and-answer function is exemplified.

[0142] The form of each of the two electronic devices can refer to the description of a single electronic device above. For example, the two electronic devices can be a mobile phone and a tablet computer, two mobile phones, or a mobile phone and a large-screen TV.

[0143] A communication connection is established between the two electronic devices, which can be based on technologies such as Bluetooth (BT), wireless local area networks (WLAN) such as wireless fidelity point to point (Wi-Fi P2P), cellular networks, remote connections based on the same system account, etc., or based on wired connection technology, which is not limited here.

[0144] like Figure 9As shown, electronic device A starts the browser and displays the page provided by the browser application on the screen. Electronic device B starts the smart assistant and displays a dialog box between the smart assistant and the user on the screen. When the user inputs a voice to the smart assistant of electronic device B, the smart assistant can answer the user's question in combination with the information displayed on the screen of electronic device A, and can also trigger electronic device A to scroll to or highlight the basis for the answer of the smart assistant in electronic device B. This part can be referred to Figure 1 and Figure 2 The difference is that electronic device A and electronic device B exchange information through a communication connection.

[0145] In some implementations, electronic device A and electronic device B may also start the same application program, in which case the two electronic devices may also exchange information.

[0146] Other implementations

[0147] Figures 1-9 Only a portion of the scenarios of dynamic information interaction is shown, such as intelligent question and answer, navigation, document analysis, table processing, and translation. Not limited to these scenarios, the intelligent interaction method of the embodiment of the present application can also be applied to other scenarios, such as text analysis (such as extracting abstracts, summarizing the main idea, etc.), content evaluation (such as painting evaluation, composition evaluation, photography evaluation, etc.), image understanding (such as talking about pictures, copywriting), drawing (such as drawing according to text), making travel plans, etc. For example, an electronic device can display text, various works, images, and text in the left area 102a of the screen, and display a dialog box of an intelligent assistant and a user in the right area 102b of the screen, so that after the user enters a question to the intelligent assistant, the intelligent assistant can answer the user in combination with the information displayed in the left area 102a of the screen. In addition, the electronic device can also make some changes (such as scrolling display or highlighting, etc.) in combination with the answer of the intelligent assistant in the information displayed in the left area 102a of the screen. The use scenario of the method provided in the embodiment of the present application depends on the actual needs of the user and is not limited here.

[0148] Different scenarios can be combined with each other, e.g. Figure 4 This example shows a scenario where intelligent question answering and navigation are combined. Another example is where translation and content evaluation can be combined.

[0149] The above embodiments can also be implemented in combination, for example Figure 1 After (b), the user can continue to input voice to the smart assistant, and the text corresponding to the voice can be Figure 2 The text 101h shown, after which the user interface displayed by the electronic device may refer to Figure 2 (b)-(c).

[0150] Figures 1-9The example shows that the user inputs voice to interact with the intelligent assistant, but it is not limited to this. The user can also input information in other modalities to interact with the intelligent assistant, or input information in multiple modalities simultaneously or sequentially to interact with the intelligent assistant. For example, the user can also input information in image modality, text module, video modality, link modality, card modality, chat history modality, document modality, etc. to the intelligent assistant. The intelligent assistant can use the content extraction technology of the corresponding modality to extract content from information in different modalities.

[0151] Figures 1-9 The example shows that the user inputs voice to the intelligent assistant to interact. In some embodiments, the intelligent assistant can proactively obtain information displayed in other areas of the screen in advance, and can predict the questions the user wants to know, and display these question options in dialog box 101 for the user to select. This can predict the user's questions and simplify the user's subsequent operations. The user only needs to select the question, without the user having to enter cumbersome voice or text to ask the intelligent assistant. The way in which the intelligent assistant predicts questions based on the information displayed in other areas of the screen is not limited. For example, it can be predicted through neural networks, large AI models, etc.

[0152] Figures 1-9 This example illustrates information exchange between the smart assistant and other applications. This isn't limited to the smart assistant; other applications can also interact. For example, a translation app can interact with an image gallery, allowing the translation app to directly translate text from an image displayed on the screen without requiring the user to enter the text into the translation app.

[0153] Figure 1-Figure 7 The example illustrates an interface where a single electronic device splits its screen left and right, displaying information from different applications in each split. In other embodiments, the single electronic device can also split the screen up and down, or split the screen into floating windows. The position of each application on the screen is not limited. For example, a smart assistant can be displayed in the right area of ​​the screen when the screen is split left and right, or in the left area of ​​the screen when the screen is split left and right, or in the upper area of ​​the screen when the screen is split up and down, etc.

[0154] Figure 1-Figure 7 This example illustrates an interface in which a single electronic device splits a screen into two parts, each displaying information from different applications. In other embodiments, a single electronic device may include multiple screens, each displaying information from different applications. This allows the information displayed on different screens to interact, providing better user experience.

[0155] Figure 8The example shows a smart question-and-answer scenario provided by a single electronic device in a dual window. However, this is not limited to the above. A single electronic device can also provide other scenarios such as navigation, document analysis, and table processing in a dual window, which are not illustrated here one by one. Not limited to dual windows, a single electronic device can also provide various scenarios in the embodiments of the present application in more windows. For example, one window displays a dialog box between the user and the smart assistant, another window displays pictures in the gallery, and another window displays a table. More windows can realize the linkage processing of information from more applications.

[0156] Figure 9 The example shows an intelligent question-and-answer scenario jointly provided by two electronic devices. However, the two electronic devices can also jointly provide other scenarios such as navigation, document analysis, and table processing, which are not illustrated here one by one. More electronic devices can also jointly provide various scenarios in the embodiments of the present application. For example, one electronic device displays a dialog box between the user and the intelligent assistant, another electronic device displays pictures from the gallery, and another electronic device displays a table. More electronic devices can realize the linkage processing of information from more applications.

[0157] Figures 1-8 shows an information interaction scenario provided by a single electronic device, Figure 9 The following illustrates information interaction scenarios provided by multiple electronic devices. In the embodiments of the present application, the above two scenarios can be implemented in combination. For example, electronic device A displays user interfaces of different applications in split screens, one for a browser and the other for an intelligent assistant, while electronic device B displays a spreadsheet document. Then, when interacting with the user, the intelligent assistant in electronic device A can combine the information displayed in the browser on this device and the information in the spreadsheet document on electronic device B to answer the user's questions.

[0158] Not limited to Figure 1-Figure 7 The electronic device shown displays information of different applications in split screens. In some embodiments, the electronic device can also display information of the same application in split screens, such as displaying information of different pages of the same application in split screens. In this case, the intelligent interaction method provided by the embodiment of the present application can also be used. For example, referring to Figure 1 , Figure 1 The page 104 displayed in the left area 102a of the screen in (b) may also be provided by the smart assistant after searching on the network.

[0159] Similar, but not limited to Figure 8 The electronic device shown displays information of different applications in separate windows. The electronic device can also display information of the same application in separate windows. In this case, the intelligent interaction method provided in the embodiment of the present application can also be used.

[0160] Similar, but not limited to Figure 9 The different electronic devices shown each display information of different applications. Different electronic devices can also each display information of the same application (actually installed on different devices) on the screen. In this case, the intelligent interaction method provided in the embodiment of the present application can also be used.

[0161] Figure 10 This is a flow chart of the intelligent interaction method provided in an embodiment of the present application. This method can be applied to a single electronic device, and the form of the single electronic device can be referred to above.

[0162] like Figure 10 As shown, the method may include the following steps:

[0163] S101 , displaying a first user interface of a first application in a first area of ​​a screen, and displaying a second user interface of a second application in a second area of ​​the screen, where the second application is different from the first application.

[0164] The embodiment of the present application does not limit the size of the first area and the second area. The first area and the second area may not overlap at all or may overlap partially.

[0165] For example, the first region may refer to Figure 1-Figure 7 The left area 102a of the middle screen, the second area can refer to the previous Figure 1-Figure 7 The right area 102b of the middle screen. The first user interface can refer to Figure 1-Figure 7 The interface displayed in the left area 102a of the middle screen, such as page 104, page 105, page 106, page 107, page 108, etc., can be referred to as the second user interface. Figure 1-Figure 7 The right area 102b of the middle screen displays a page containing the dialog box 101.

[0166] Alternatively, the first region may refer to Figure 8 The second area can refer to the above Figure 8 Window 130 in the screen.

[0167] The first application and the second application are different application programs. The first application can be, for example, a browser, a map application, a gallery, a spreadsheet application, etc., and the second application can be, for example, an intelligent assistant.

[0168] In some embodiments, when the electronic device has only one screen, the first area and the second area are two areas obtained by the electronic device performing split-screen display on the screen. Specifically, before displaying the first user interface of the first application in the first area of ​​the screen and displaying the second user interface of the second application in the second area of ​​the screen, the electronic device may display the first user interface of the first application and the second user interface of the second application in split-screen display on the screen.

[0169] The situations where electronic devices display split screens include the following:

[0170] 1. Before the electronic device displays a first user interface of a first application in a first area of ​​the screen and displays a second user interface of a second application in a second area of ​​the screen, it receives a user input to display the first and second applications in a split-screen mode. This is equivalent to the user inputting the split-screen operation, thereby triggering the electronic device to split the screen.

[0171] 2. The electronic device first displays a second user interface on the screen and receives a second question input in the second user interface, where the second question is related to the first application; then, in response to the input of the second question, the electronic device opens the first user interface of the first application in the first area of ​​the screen and displays the second user interface of the second application in the second area of ​​the screen.

[0172] The second user interface displayed on the screen of the electronic device can refer to Figure 1 (a) shows the user interface, Figure 3 (a) shows the user interface, Figure 4 (a) shows the user interface. For the second question, please refer to Figure 1 (a) shows the text 101c, Figure 3 (a) shows the text 101j, Figure 4 (a) shows the text 101n.

[0173] In the second case, it is equivalent to the second application triggering the electronic device to launch the first application in response to the input of the second question. The second question is related to the first application, which means that the first application can solve the second question. For example, when the second question involves navigation, the first application is a map application, or when the second question involves translation, the first application is a translation application. The intelligent assistant can analyze the second question and the various applications installed in the electronic device to determine the first application and trigger its launch.

[0174] 3. The electronic device first displays a second user interface on the screen and receives a first document and a third question entered in the second user interface, where the third question is related to the first document. Then, in response to the input of the first document and the third question, the electronic device opens a first user interface of a first application in a first area of ​​the screen and displays a second user interface of a second application in a second area of ​​the screen, where the first user interface is the interface for displaying the first document in the first application. The order in which the first document and the third question are entered is not restricted; they can be entered simultaneously, or the first document can be entered first, or the third question can be entered first.

[0175] The second user interface displayed on the screen of the electronic device can refer to Figure 5 (a) shows the user interface, the first document can refer to Figure 5(a) shows the PDF document. For the third question, please refer to Figure 5 (a) shows a text 101t.

[0176] In the third case, it is equivalent to the second application triggering the electronic device to launch the first application in response to the input of the first document and the third question.

[0177] In the above three cases, the split screen style of the electronic device can be left-right split screen, top-bottom split screen, floating window split screen, etc., which is not limited here.

[0178] In other embodiments, when the number of screens of the electronic device is one, the first area and the second area are two areas of the electronic device that are displayed in separate windows. Specifically, the first area is the display area of ​​the first window on the screen, and the second area is the display area of ​​the second window on the screen. For example, the first window may refer to Figure 8 In window 109, the first area can refer to Figure 8 The second window can refer to the area of ​​window 109 in Figure 8 In the window 130, the second area can refer to Figure 8 The area of ​​window 130 in FIG.

[0179] In other embodiments, when the electronic device has multiple screens, for example, when the electronic device includes a first screen and a second screen, the first area is the display area of ​​the first screen, and the second area is the display area of ​​the second area. In this way, the electronic device can present information of different applications on different screens, and the information presented on different screens can also interact, providing better services to users.

[0180] S102: Receive a first operation, where the first operation includes an operation acting on the second area and does not include an operation acting on the first area.

[0181] In some embodiments, the first operation includes a first question input by the user. The first question can be one of the following modes or a combination of multiple modes: voice, text, image, video, link, card, chat history, document. When the mode of the first question includes voice mode, the electronic device can display a second text in the second user interface in response to the first question, and the second text is the text converted from the first question in voice mode. The second text can refer to Figure 2 Text 101h, Figure 3 Text 101L, Figure 4 Text 101n or Text 101p or Text 101r, Figure 5 Text 101t, Figure 6 Text 101w or text 101y, Figure 7 Text 101A.

[0182] In other embodiments, before receiving the first operation, the electronic device may display one or more question options in the second user interface, where one question option corresponds to a question, and the questions corresponding to the one or more question options are determined based on the information presented in the first user interface. Figure 5 In this embodiment, the first operation includes an operation on a first question option among the one or more question options. The first question option may be, for example, an option to interpret a document, an option to read the full text aloud, an option to translate the full text, an option to extract text, etc.

[0183] S103: In response to the first operation, display first information in the second user interface, where the first information is determined based on information presented in the first user interface.

[0184] The information presented by the first user interface may include the information displayed on the screen by the first user interface at that moment, or may include the information displayed on the screen by the first user interface at that moment and other information that may be visible when a page is subsequently turned.

[0185] The electronic device can use the second application to take a screenshot of the first user interface to obtain a first screenshot, and obtain information presented by the first user interface from the first screenshot; or, through the second application, obtain information presented by the first user interface from the operating system of the electronic device.

[0186] For example, the first information may refer to Figure 2 Text 101i, Figure 3 Text 101m, Figure 4 Text 101o or Text 101q or Text 101s, Figure 5 Text 101u, Figure 6 Charts 101x or Text 101z, Figure 7 Text 101B.

[0187] In some embodiments, the first operation includes inputting a first question. Before displaying the first information in the second user interface, the electronic device may determine whether the first question contains a preset designated word, or determine whether the first question is relevant to the information presented in the first user interface. In this way, the electronic device determines the first information based on the information presented in the first user interface.

[0188] Through step S103, the first information displayed in the second area is determined according to the information in the first area, that is, the information between the two applications in the electronic device can dynamically interact to provide the user with a good usage experience.

[0189] S104: Refresh the content displayed in the first area in response to the first operation, or in response to a second operation performed by the user on the second user interface after the first operation.

[0190] The electronic device can refresh the content displayed in the first area under the triggering of the first operation or under the triggering of the second operation. The second operation is different from the first operation. The form of the second operation can refer to the first operation and will not be repeated here.

[0191] The electronic device refreshes the content displayed in the first area in the following two ways:

[0192] 1. The electronic device refreshes the information displayed by the first application in the first area.

[0193] Mode 1 is equivalent to the electronic device still displaying the information of the first application in the first area. Specifically, the electronic device may refresh the information of the first application in the first area in the following ways:

[0194] (1) The electronic device scrolls to display the first basis in the first user interface, or highlights the first basis in the first user interface.

[0195] The electronic device determines the first information based on a first basis in the information presented by the first user interface, that is, the first basis is to determine the key content of the first information. The first basis can be in the form of text, chart, image, video, link, card, chat record, etc.

[0196] For an example of the first basis, please refer to Figure 2 The basis for 104b, Figure 4 The method of highlighting the first basis can be referred to the above description.

[0197] In some embodiments, the electronic device may transmit the first basis information (for example, the first information in the preceding text) from the second application to the first application, so that the first application scrolls to display the first basis or highlights the first basis in the first user interface.

[0198] In other embodiments, the second application in the electronic device can highlight the first basis in the first user interface through the operating system without the first application being aware of it. For example, before rendering, the operating system of the electronic device can retrieve the first basis to be highlighted in the first user interface, then add a layer mask or other highlighting style to the first basis, and then render the first user interface and display the rendered result on the screen, thereby highlighting the first basis in the first user interface.

[0199] Through the first method, the electronic device can scroll or highlight the basis of the answer on the right side in the first user interface, which allows the user to easily view the basis and also shows that the answer of the smart assistant has a basis, which can enhance the user's trust in the smart assistant and provide a better interactive experience between the user and the smart assistant.

[0200] (2) The electronic device adds second information in the first user interface, and the second information is determined based on the information presented in the first user interface.

[0201] For an example of the second information, please refer to Figure 3 The location of the gas station shown in (c) Figure 6 Fill in the average value column in the table from (c). Figure 7 The second method allows the electronic device to add some new information to the first user interface, thereby providing more information to the user.

[0202] In some embodiments, the electronic device can transmit the second information from the second application to the first application, so that the first application displays the second information in the first user interface. In other embodiments, the second application in the electronic device can display the second information in the first user interface through the operating system without the first application being aware of it, for example, by adding a layer on top of the first user interface to display the second information.

[0203] The second method allows the information of the two applications to further interact by backfilling the content into the first user interface.

[0204] (3) The electronic device displays the fourth user interface of the first application in the first area.

[0205] Method (3) is equivalent to the electronic device switching the interface of the first application to the fourth user interface in the first area. The fourth user interface and the first user interface have different functions. The function of the fourth user interface may be related to the first operation. Method (3) allows the electronic device to switch to the user interface of other functions within the same APP, thereby providing users with more functions.

[0206] For example, refer to Figure 4 (b) and (c), when the second operation received by the electronic device is the user input voice "What special food does this 'restaurant 1' have", the electronic device responds to the second operation and switches the page displayed by the food recommendation APP in the left area 102a of the screen. Figure 4 The page of the food recommendation APP shown in (b) is switched to Figure 4Another page of the same APP shown in (c) is used to display nearby restaurants, and the function of the latter page is to display detailed information of "Restaurant 1". Figure 4 The page of the food recommendation APP shown in (c) is the details page of "Restaurant 1", which is related to the voice input by the user "What special foods does this 'Restaurant 1' have?"

[0207] 2. The electronic device displays a third user interface of a third application in the first area, where the third application is different from the first application.

[0208] Mode 2 is equivalent to the electronic device jumping to display the user interface of another application in the first area, thereby providing the user with the function of another application. Figure 4 (c) and (d), when the second operation received by the electronic device is the user input voice "Navigate to 'Restaurant 1'", the electronic device responds to the second operation and switches from page 106 of the food recommendation APP to page 107 of the map APP in the first area.

[0209] The above describes several ways to refresh the information of the first application in the first area. In the embodiment of the present application, the electronic device can perform a corresponding refresh action in the first area of ​​the electronic device based on the actual needs of the user (such as the first operation or the second operation). When the user's needs are different, the electronic device can perform different refresh actions.

[0210] S105 , displaying third information in the second user interface, where the third information indicates the basis of the first information in the first user interface.

[0211] The third information can be implemented as text and / or controls. Figure 5 (b) shows text 101v. A link is a form of control. When the third information is implemented as a link, the link points to the basis of the first information. The electronic device can receive a user operation on the link and then, in response to the user operation, open the basis of the first information corresponding to the link in the first area. Alternatively, it can also open the basis of the first information corresponding to the link in the second area in response to the user operation.

[0212] S105 can indicate the basis in the second user interface to indicate that the smart assistant's answer is based on evidence, which can enhance the user's trust in the smart assistant and provide a better interactive experience between the user and the smart assistant.

[0213] S104 and S105 are both optional operations, and the embodiment of the present application does not limit the order of these two operations.

[0214] pass Figure 10In the method shown, when an electronic device displays information from different applications in different areas on the screen, it can analyze the information in the first area and display the results of the analysis in the second area. In this way, the information of two applications displayed simultaneously by the electronic device can break down barriers and achieve dynamic information interaction. In addition, the analysis results can be fed back to the first area to refresh the content displayed in the first area, which further strengthens the information interaction between the two applications. This can provide users with an intelligent interaction method, facilitating efficient interaction between users and electronic devices.

[0215] Figure 11 This is a flow chart of another intelligent interaction method provided in an embodiment of the present application. This method can be applied to multiple electronic devices that have established communication connections.

[0216] Figure 11 The process of this method is illustrated by taking two electronic devices as an example. The configurations of the two electronic devices can be referred to above. Figure 11 As shown, the method may include the following steps:

[0217] S201: A first device displays a first user interface of a first application on a first screen.

[0218] S202: The second device displays a second user interface of a second application on a second screen, where the second application is different from the first application.

[0219] S203: Receive a first operation, where the first operation includes an operation acting on the second screen and does not include an operation acting on the first screen.

[0220] S204: The second electronic device obtains information presented by the first user interface from the first device through a communication connection.

[0221] The first device may obtain the information presented by the first user interface by taking a screenshot or obtaining the information from the OS, and then send the information presented by the first user interface to the second device through a communication connection.

[0222] S205: The second device displays first information in the second user interface in response to the first operation, where the first information is determined based on the information presented in the first user interface.

[0223] The first device can refer to Figure 9 The electronic device A shown, the second device can refer to Figure 9 The electronic device B shown. The first user interface can refer to Figure 9 The user interface 120 shown, the second user interface can refer to Figure 9 The user interface 140 is shown. The first operation may be, for example, Figure 9 The voice input shown in the figure is "Huawei and Avita Investment Agreement, how much shares do they hold?". The first information can be, for example, Figure 9 The text displayed reads: "According to news on August 20, Avita holds a 10% stake."

[0224] Figure 11 The two electronic devices shown perform an intelligent interaction method, and Figure 10 The interaction method of the single electronic device shown is similar, except that Figure 11 The method involves interaction between two electronic devices and requires information to be transmitted between the two electronic devices. Figure 10 The method involves only one electronic device and only requires information to be transferred between different applications without requiring information to be transferred between devices. Figure 10 Optional steps, optional implementation methods, etc. of the method, Figure 11 The method is also applicable, with the only difference being the above. Therefore, this application embodiment will not explain these optional steps and optional implementation methods in detail. Please refer to Figure 10 Related introduction.

[0225] pass Figure 11 In the method shown, multiple electronic devices can display information from different applications, and the second device can analyze the information displayed by the first device, and the results of the analysis are displayed on the second device. In this way, the information of the application displayed simultaneously by two electronic devices can break down barriers and complete dynamic information interaction. In addition, the results of the analysis can also be fed back to the first device, for example, the first device can scroll to or highlight the basis for the results, or backfill the results in the first device, etc., which further strengthens the information interaction between the two devices. This can provide users with an intelligent interaction method, facilitating efficient interaction between users and electronic devices.

[0226] Figure 12 This is a hardware structure diagram of an electronic device provided in an embodiment of the present application. The electronic device may be any of the electronic devices mentioned in the above embodiments.

[0227] like Figure 12 As shown, the electronic device may include: the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a camera 193, a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180K, etc.

[0228] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0229] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0230] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0231] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0232] The charging management module 140 is configured to receive charging input from a charger, which may be a wireless charger or a wired charger.

[0233] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160.

[0234] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0235] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0236] Mobile communication module 150 can provide wireless communication solutions for electronic devices, including 2G / 3G / 4G / 5G. Mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. Mobile communication module 150 can receive electromagnetic waves from antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. Mobile communication module 150 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via antenna 1.

[0237] The modem processor includes a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a medium- or high-frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is passed to the application processor.

[0238] The wireless communication module 160 can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0239] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that the electronic device can communicate with the network and other devices through wireless communication technology.

[0240] The electronic device implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0241] Display screen 194 is used to display images, videos, etc. Display screen 194 is the screen mentioned above. Display screen 194 includes a display panel. In some embodiments, the electronic device may include one or N display screens 194, where N is a positive integer greater than one. When the electronic device is a non-folding screen device, it may include one display screen 194. When the electronic device is a folding screen device, it may include a whole foldable display screen 194, or it may include multiple display screens 194, and some or all of the multiple display screens 194 may be combined into one display screen.

[0242] The electronic device can realize the shooting function through the ISP, camera 193, video codec, GPU, display 194 and application processor.

[0243] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in electronic devices, such as image recognition, face recognition, speech recognition, and text comprehension.

[0244] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0245] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0246] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0247] The external memory interface 120 can be used to connect to an external non-volatile memory device to expand the storage capacity of the electronic device. The external non-volatile memory device communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored in the external non-volatile memory device.

[0248] The electronic device can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0249] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, the electronic device detects the touch intensity based on pressure sensor 180A. The electronic device can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, an instruction to create a new short message is executed.

[0250] The touch sensor 180K is also referred to as a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device, at a location different from that of the display screen 194.

[0251] In an embodiment of the present application, a computer program for implementing the intelligent interaction method provided in an embodiment of the present application is stored in the internal memory 121, and the processor 110 can be used to execute the computer program to implement the method provided in an embodiment of the present application.

[0252] When the method provided in the embodiment of the present application is applied to a single electronic device, the display screen 194 can be used to display the aforementioned Figures 1-8 The user interface shown. The display screen 194 is used to receive text input by the user, and can also be used to receive tables, pictures and other information selected by the user locally. The microphone 170C is used to receive voice input by the user. The processor 110 is used to run various applications. When the processor 110 runs the intelligent assistant, it can identify various modal questions input by the user, and answer the user's questions in combination with information other than the intelligent assistant on the screen. It can also mobilize the display screen to highlight the basis for the answer in the display area other than the intelligent assistant, or display backfill information such as translation, calculation results of table data, etc. The operations performed by each component in the electronic device can also be referred to in the previous text. Figure 10 The description of the method embodiment shown will not be expanded in detail here.

[0253] When the method provided in the embodiment of the present application is applied to multiple electronic devices, Figure 12 The electronic device shown may be one of the plurality of electronic devices. The display screen 194 may be used to display the aforementioned Figure 9 The user interface displayed by the electronic device A or electronic device B shown. If the electronic device is the electronic device B mentioned above, the display screen 194 is used to receive text input by the user, and can also be used to receive tables, pictures and other information selected locally by the user, and the microphone 170C is used to receive voice input by the user; the processor 110 can run an intelligent assistant to identify questions of various modes input by the user, and answer the user's questions in combination with the information displayed on the screen of the electronic device A. It can also mobilize the screen of the electronic device A to highlight the basis for the answer, or display backfill information such as translation, calculation results of table data, etc. If the electronic device is the electronic device A mentioned above, the display screen 194 is used to highlight the basis for the answer on the screen, or display backfill information such as translation, calculation results of table data, etc. The communication modules of the electronic device, such as antenna 1, antenna 2, mobile communication module 150, and wireless communication module 160, are also responsible for communicating with other electronic devices (such as electronic device A or electronic device B) to exchange information. The operations performed by each component in the electronic device can also be referred to above. Figure 11 The description of the method embodiment shown will not be expanded in detail here.

[0254] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes a mobile operating system with a layered architecture as an example to illustrate the software structure of the electronic device.

[0255] Figure 13 A software structure diagram of the electronic device provided in an embodiment of the present application.

[0256] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, a mobile operating system is divided into four layers: the application layer, the framework layer / core services layer, the system libraries and runtime layer, and the kernel layer.

[0257] The application layer can include a series of application packages.

[0258] like Figure 13 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0259] The program framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The program framework layer includes some predefined functions.

[0260] like Figure 13 As shown, the program framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0261] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0262] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0263] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0264] The phone manager is used to provide communication functions for electronic devices, such as call status management (including answering, hanging up, etc.).

[0265] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0266] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0267] The runtime can refer to all code libraries, frameworks, and other components required for a program to run. For example, for the C language, the runtime includes a series of function libraries required for C programs to run. For the Java language, in addition to the core libraries, the runtime also includes the virtual machine required for Java programs to run. These core libraries include the functions required by the Java language.

[0268] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0269] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0270] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0271] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0272] A 2D graphics engine is a drawing engine for 2D drawings.

[0273] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0274] The following describes the workflow of the electronic device software and hardware in conjunction with capturing a photo scene.

[0275] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a still image or video through the camera 193.

[0276] It should be understood that each step in the above method embodiment can be completed by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0277] The present application also provides a communication system, comprising a first device and a second device, wherein the first device is configured to execute Figure 11 The operation performed by the first device in the method shown is performed by the second device Figure 11 The operations performed by the second device in the method shown.

[0278] The present application also provides an electronic device, which may include a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement a method executed on the electronic device side in any of the above-mentioned single electronic device embodiments, or a method executed on the first device side or the second device side in the above-mentioned multiple device embodiments.

[0279] The present application also provides a chip system, including a processing circuit and an interface circuit, the interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to run the computer instructions to implement a method executed on the electronic device side in any of the above-mentioned single electronic device embodiments, or a method executed on the first device side or the second device side in the above-mentioned multiple device embodiments.

[0280] The present application also provides a chip system, which includes at least one processor for implementing the method executed on the electronic device side in any of the above-mentioned single electronic device embodiments, or the method executed on the first device side or the second device side in the above-mentioned multiple device embodiments. In one possible design, the chip system also includes a memory for storing program instructions and data, and the memory is located within or outside the processor.

[0281] A chip system can be composed of chips or include chips and other discrete devices.

[0282] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0283] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0284] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0285] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the method executed on the electronic device side in any of the above-mentioned embodiments of a single electronic device, or the method executed on the first device side or the second device side in the above-mentioned embodiments of multiple devices.

[0286] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements a method executed on the electronic device side in any of the above-mentioned single electronic device embodiments, or a method executed on the first device side or the second device side in the above-mentioned multiple device embodiments.

[0287] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.

[0288] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0289] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0290] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0291] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0292] In short, the above description is only an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of this application should be included in the scope of protection of this application.

Claims

1. An intelligent interaction method, characterized in that: Applied to an electronic device including a screen, the method includes: displaying a first user interface of a first application in a first area of ​​the screen and displaying a second user interface of a second application in a second area of ​​the screen, wherein the first application is different from the second application; receiving a first operation, where the first operation includes an operation acting on the second application and does not include an operation acting on the first application; In response to the first operation, first information is displayed in the second user interface, where the first information is determined based on information displayed in the first user interface.

2. The method according to claim 1, characterized in that The method further comprises: In response to the first operation, or in response to a second operation on the second user interface after the first operation, content displayed in the first area is refreshed.

3. The method according to claim 2, characterized in that Refreshing the content displayed in the first area specifically includes: Refreshing information displayed by the first application in the first area; or, A third user interface of a third application is displayed in the first area, where the third application is different from the first application.

4. The method according to claim 3, characterized in that Refreshing the information displayed by the first application in the first area specifically includes: Scrolling to display a first basis in the first user interface, the first information is determined according to the first basis; Alternatively, highlighting the first basis in the first user interface; Alternatively, second information is added to the first user interface, and the second information is determined based on the information presented in the first user interface; Alternatively, a fourth user interface of the first application is displayed in the first area.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Third information is displayed in the second user interface, where the third information is used to indicate a basis for the first information in the first user interface.

6. The method according to any one of claims 1 to 5, characterized in that The first operation includes inputting a first question. Before displaying the first information in the second user interface, the method further includes: Determining that the first question contains a preset designated word; Alternatively, it is determined that the first question is related to the information presented by the first user interface.

7. The method according to claim 6, characterized in that The first question input includes one of the following modalities or a combination of multiple modalities: voice, text, image, video, link, card, chat history, document.

8. The method according to any one of claims 1 to 5, characterized in that Before receiving the first operation, the method further includes: Displaying one or more question options in the second user interface, where each question option corresponds to a question, and the questions corresponding to the one or more question options are determined based on the information presented in the first user interface; The first operation includes an operation acting on a first question option among the one or more question options.

9. The method according to any one of claims 1 to 8, characterized in that The number of the screen is one, and a first user interface of a first application is displayed in a first area of ​​the screen, and a second user interface of a second application is displayed in a second area of ​​the screen, specifically including: The first user interface of the first application and the second user interface of the second application are displayed in split screen on the screen.

10. The method according to claim 9, characterized in that Before displaying the first user interface of the first application in the first area of ​​the screen and displaying the second user interface of the second application in the second area of ​​the screen, the method further includes: A user operation of displaying the first application and the second application in split screen is received.

11. The method according to claim 9, characterized in that Before displaying the first user interface of the first application in the first area of ​​the screen and displaying the second user interface of the second application in the second area of ​​the screen, the method further includes: displaying the second user interface on the screen; receiving a second question input in the second user interface, where the second question is related to the first application; The first area of ​​the screen displays the first user interface of the first application, and the second area of ​​the screen displays the second user interface of the second application, specifically including: in response to the input of the second question, opening the first user interface of the first application in the first area of ​​the screen, and displaying the second user interface of the second application in the second area of ​​the screen.

12. The method according to claim 9, characterized in that Before displaying the first user interface of the first application in the first area of ​​the screen and displaying the second user interface of the second application in the second area of ​​the screen, the method further includes: displaying the second user interface on the screen; receiving a first document and a third question input in the second user interface, wherein the third question is related to the first document; The first area of ​​the screen displays the first user interface of the first application, and the second area of ​​the screen displays the second user interface of the second application, specifically including: in response to the input of the first document and the third question, opening the first user interface of the first application in the first area of ​​the screen, and displaying the second user interface of the second application in the second area of ​​the screen, wherein the first user interface is an interface for displaying the first document in the first application.

13. The method according to any one of claims 1 to 8, characterized in that The number of the screen is one, the first area is an area of ​​a first window in the screen, and the second area is an area of ​​a second window in the screen.

14. The method according to any one of claims 1 to 8, characterized in that The number of the electronic device is one, the screen includes a first screen and a second screen, the first area is a display area of ​​the first screen, and the second area is a display area of ​​the second screen.

15. The method according to any one of claims 1 to 8, characterized in that The electronic device includes: a first device having a first screen, and a second device having a second screen. The first area is a display area of ​​the first screen, and the second area is a display area of ​​the second screen.

16. The method according to any one of claims 1 to 15, characterized in that Before displaying the first information in the second user interface, the method further includes: Taking a screenshot of the first user interface to obtain a first screenshot, and obtaining information presented by the first user interface from the first screenshot; or, Information presented by the first user interface is obtained from the operating system of the electronic device.

17. The method according to any one of claims 1 to 16, characterized in that The second application is a smart assistant application.

18. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

20. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

21. A chip system, comprising a processing circuit and an interface circuit, wherein the interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to execute the computer instructions to implement the method according to any one of claims 1 to 17.

Images