Interaction method and electronic equipment

Abstract

The invention discloses an interaction method and electronic equipment, and the electronic equipment can select an interaction object after detecting a specific operation (such as a long-pressing operation for a physical key on a mouse) of a user for equipment (such as the mouse) which is connected with the electronic equipment and has a cursor attribute, and starts an intelligent interaction function. According to the electronic equipment, the cursor is displayed on the electronic equipment, the intelligent interaction entrance is displayed at the position where the cursor is located or near the cursor, after it is detected that a user inputs an instruction, the electronic equipment can display the instruction at the intelligent interaction entrance, and after the instruction is input, the electronic equipment can substitute an interaction object into the input instruction for understanding and execution. Therefore, the task can be completed without moving the cursor for a long distance, in addition, the specific operation can trigger the electronic equipment to recognize the content of the interaction object and provide intelligent menu options based on the content, so that the quick operation for the interaction object is completed, the operation is convenient, quick and efficient, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an interaction method and electronic device. Background Art

[0002] With the development of electronic devices, more and more electronic devices (such as personal computers, etc.) can provide natural language interaction functions, such as voice interaction functions. The voice interaction function can support electronic devices to receive voice commands input by users, and understand and execute the voice commands.

[0003] However, the natural language interaction entrance on the current personal computer (PC) is usually displayed at a fixed position on the screen. If the user wants to operate an interactive object through the natural language interaction function, the user needs to first select the interactive object and move the cursor to the natural language interaction entrance to perform the corresponding operation. After the operation is completed, the cursor is moved back to the original position. Such frequent movement of the cursor cannot bring an intelligent experience to the user, and the task execution efficiency is also low. If some tasks have more operation steps, the cursor needs to make a lot of meaningless movements on the screen, and the moving distance is also long, which is complicated and cumbersome, and the user experience is poor. Summary of the invention

[0004] An embodiment of the present application provides an interaction method and an electronic device, which can display an intelligent interaction entrance at or near the cursor location and support users to operate the interaction object through the intelligent interaction entrance, thereby solving the problem of moving the cursor over long distances and improving human-computer interaction efficiency and user experience.

[0005] In a first aspect, an embodiment of the present application provides an interaction method, which is applied to an electronic device, wherein the electronic device includes a screen, and the method includes: the electronic device detects a first operation; in response to the first operation, the electronic device turns on an intelligent interaction function and displays an intelligent interaction entrance, wherein the intelligent interaction function is used for the electronic device to interact with a user through voice commands or text commands, wherein the intelligent interaction entrance is displayed at or near a first position, and the first position is the position of a cursor on the screen when the first operation is detected; the electronic device detects that a user inputs a first instruction through the intelligent interaction entrance, and determines that the interaction object is a first interaction object based on the first instruction; the electronic device executes the operation corresponding to the first instruction for the first interaction object.

[0006] Among them, the first instruction can be a voice instruction or a text instruction input by the user, and the first interactive object can be an object used for intelligent interaction, such as pictures, videos, texts, files, applications or application icons, cards and other objects.

[0007] By implementing the method provided in the first aspect, the electronic device can support the user to enable the intelligent interaction function through the first operation and display the intelligent interaction entrance at or near the cursor position, and can support the user to operate the interactive object through the intelligent interaction entrance. In this way, the task can be completed without the user moving the cursor over a long distance to a fixed intelligent interaction entrance, which is convenient and fast, and improves the human-computer interaction efficiency and user experience.

[0008] In some implementations, the intelligent interaction portal may be a natural language interaction portal, such as an artificial intelligence interaction portal, a voice recognition interaction portal, etc. The intelligent interaction portal is not limited in this application.

[0009] In a possible implementation manner, the electronic device is connected to an accessory device, and the accessory device is used to receive the first operation and an operation of moving a cursor by a user.

[0010] Among them, the accessory device can be, for example, a device with cursor attributes such as a mouse and a touchpad, and the first operation can be an operation performed by the user on the accessory device to trigger the electronic device to activate the intelligent interaction function and display the intelligent interaction entrance at or near the cursor position.

[0011] In this way, the user can trigger the electronic device to turn on the smart interaction function and display the smart interaction entrance at or near the cursor position through a one-step operation on the accessory device. The user does not need to move the cursor over a long distance to a fixed smart interaction entrance. This is convenient and quick, and improves the efficiency of human-computer interaction.

[0012] In a possible implementation, the accessory device is a mouse, the mouse includes a left button and a right button, and the first operation is an operation of long pressing the left button, or an operation of long pressing the right button, or an operation of long pressing both the left button and the right button.

[0013] In this way, there is no need to add additional physical buttons on the mouse to trigger the electronic device to start the intelligent interaction function and display the intelligent interaction entrance at or near the cursor position, saving development costs.

[0014] It can be understood that the first operation is not limited to the above three long press operations, and may also be other operations performed by the user on the left key and / or the right key.

[0015] In some embodiments, a physical button may be added to the mouse, and the user's corresponding operation (such as a click operation) on the physical button may also trigger the electronic device to activate the intelligent interaction function and display an intelligent interaction entrance at or near the cursor position.

[0016] In one possible implementation, the electronic device determines that the interactive object is the first interactive object based on the first instruction, specifically including: when the first instruction does not include the first keyword, the first keyword is a keyword used to indicate the interactive object, and the electronic device determines that the first interactive object is an identifiable object at the first position; or, when the first instruction includes the first keyword, the electronic device determines that the interactive object indicated by the first keyword is the first interactive object.

[0017] The first keyword may be a word used to refer to an interactive object (such as "this", "it", "that", etc.), or a word used to indicate the name of an interactive object.

[0018] That is to say, after the first instruction is input, the electronic device can determine whether there is a default object in the first instruction (that is, the first instruction does not include words used to indicate the name of the interactive object) or a pronoun (that is, the first instruction includes words such as this, that, it, etc. used to refer to the interactive object). If so, the electronic device can determine the interactive object based on the first position; if not, the electronic device can determine that the interactive object indicated by the first keyword is the first interactive object, and can perform the corresponding operation based on the first instruction.

[0019] In a possible implementation, the intelligent interaction entrance is formed by the deformation of the cursor.

[0020] In this way, the electronic device can display the smart interaction entrance at the position of the cursor before deformation, instead of displaying the smart interaction entrance at a fixed position on the screen. Wherever the cursor is, the smart interaction entrance is there, which can solve the problem of moving the cursor over long distances. In addition, the process of the cursor deforming into a smart interaction entrance can provide users with a more flexible and vivid visual experience.

[0021] In a possible implementation, after the electronic device turns on the intelligent interaction function, the method further includes: the electronic device outputs a first prompt, where the first prompt is used to prompt the user to input a recognizable instruction corresponding to the first interactive object.

[0022] In this way, when the user does not know how to perform intelligent interaction, the electronic device can output corresponding prompts (such as voice prompts or text prompts) so that the user can quickly learn how to perform intelligent interaction, thereby improving interaction efficiency and user experience.

[0023] In a possible implementation, after the electronic device detects that the user inputs the first instruction, the method further includes: if the electronic device detects that the user cancels the first operation, the electronic device no longer detects the instruction input by the user.

[0024] In this way, after the user cancels the first operation, the electronic device can determine that the user has completed inputting the instruction, and can thus execute the next operation.

[0025] In a possible implementation, the first instruction does not include a keyword directly indicating the name of the interactive object.

[0026] Since the object that currently needs to be interacted with has been determined through the first operation, or the first operation and other identification operations, the user does not need to overly consider how to clearly describe the location or name of the object when entering instructions so that the electronic device can accurately identify it. The user only needs to verbally use some words that indirectly refer to the object, such as "this" and "that", or even directly describe the instructions to be operated without indicating the object. In this way, users are prevented from spending too much time describing an object that is difficult to describe, which improves the convenience of users' natural language instruction interaction and helps to increase the degree of users' use of natural language instructions.

[0027] In one possible implementation, the cursor is a pointer cursor, the first interactive object is a picture or a first content in the picture, the first content is a portion of the content in the picture, and the electronic device performs an operation corresponding to a first instruction on the first interactive object, specifically including: when the first instruction is used to indicate an intention to perform a second operation on the first interactive object, the electronic device performs a second operation on the first interactive object, wherein the second operation includes any one of the following operations: sharing, collecting, copying, erasing, replacing, inserting, beautifying, identifying, style migration, parameter adjustment, and saving.

[0028] The first picture may be a picture in an application on the electronic device, and the first content may be a part of the content in the first picture (eg, a cutout subject).

[0029] In this way, through the intelligent interactive function, electronic devices can easily and quickly perform various operations on pictures.

[0030] In one possible implementation, when the second operation is a save operation, the electronic device performs the second operation on the first interactive object, specifically including: the electronic device saves the first interactive object to a target location, where the target location is a default storage location, or the target location is determined based on a keyword included in the first instruction for indicating the storage location.

[0031] In this way, when the user wants to save the first interactive object to a certain storage location, the user does not need to perform multi-level menu jump operations. The user only needs to enter a command and the electronic device can save the first interactive object to the corresponding storage location, thereby simplifying user operations and improving human-computer interaction efficiency and user experience.

[0032] In a possible implementation, the cursor is a pointer cursor, and the first interaction object is a video. The electronic device performs an operation corresponding to the first instruction on the first interaction object, specifically including: when the first instruction is used to indicate the intention of performing a third operation on the video, the electronic device performs the third operation on the video, where the third operation includes any one of the following operations: sharing, favoriting, playing, cropping, and saving.

[0033] In this way, through the intelligent interaction function, the electronic device can conveniently and quickly perform various operations on the video.

[0034] In a possible implementation, the cursor is a pointer cursor, and the first interaction object is a first text. The electronic device performs an operation corresponding to the first instruction on the first interaction object, specifically including: when the first instruction is used to indicate the intention of summarizing the first text, the electronic device summarizes the first text to generate a second text and displays a first window, where the first window includes the second text; or, when the first instruction is used to indicate the intention of translating the first text, the electronic device translates the first text to generate a third text and displays a second window, where the second window includes the third text; or, when the first instruction is used to indicate the intention of querying the first text, the electronic device queries the first text and displays a query result, where the query result includes relevant information about the first text.

[0035] Among them, the first window can be, for example, Figure 2D the window 220 shown in the figure. At this time, the first text can be, for example, Figure 2B the selected text shown in the figure; the second window can be, for example, Fig.5I the window 520 shown in the figure. At this time, the first text can be, for example, Figure 5G the selected text shown in the figure; the query result can be, for example, Fig. 6E the search result 615 shown in the figure. At this time, the first text can be "Huizhou".

[0036] In this way, through the intelligent interaction function, the electronic device can conveniently and quickly perform various operations on the text.

[0037] In a possible implementation, the method further includes: the electronic device detects a fourth operation; in response to the fourth operation, the electronic device replaces the first text with the second text and displays the second text; or, the electronic device detects a fifth operation; in response to the fifth operation, the electronic device replaces the first text with the third text and displays the third text.

[0038] Among them, the fourth operation can be, for example, the operation of the user on Figure 2D the option 223 shown in the figure (such as a click operation), and the fifth operation can be, for example, the operation of the user on Fig.5I the option 523 shown in the figure (such as a click operation).

[0039] In this way, the electronic device can support the user to replace the text conveniently and quickly.

[0040] In one possible implementation, the cursor is a pointer cursor, and the electronic device performs an operation corresponding to a first instruction on a first interactive object, specifically including: when the first instruction is used to indicate an intention to search for a first object, the electronic device searches for the first object and displays search results, the search results including information about one or more first objects, and the type of the first object includes any one of the following: picture, video, document.

[0041] In this way, through the intelligent interaction function, the electronic device can easily and quickly implement object search operations without the need for users to manually perform complex and tedious search operations.

[0042] In one possible implementation, before the electronic device searches for the first object, the method also includes: the electronic device displays a first user interface, which is a user interface corresponding to the first document; the electronic device displays search results, specifically including: the electronic device displays the search results on the first user interface; after the electronic device displays the search results, the method also includes: the electronic device detects a sixth operation; in response to the sixth operation, the electronic device inserts the first object into the first document.

[0043] The first user interface may be, for example, the user interface 310, the first document may be, for example, a document edited by a user, and the sixth operation may be an operation performed by the user on the first object displayed in the search results (for example, the user performs an operation on Figure 3C The operation of the picture 321 shown), the first object may be, for example, the picture 321.

[0044] In this way, the electronic device can directly display the search results in the document, and the user can directly select an object from the search results and insert it into the document, which is convenient and fast and simplifies user operations.

[0045] In one possible implementation, the first object is the second document. Before the electronic device searches for the first object, the method also includes: the electronic device displays the desktop; the electronic device displays the search results, specifically including: the electronic device displays the search results on the desktop; after the electronic device displays the search results, the method also includes: the electronic device detects a seventh operation; in response to the seventh operation, the electronic device opens the second document.

[0046] The second document may be one or more documents edited by the user, such as Figure 4C The seventh operation may be an operation of the user on the document item displayed in the search results (for example, the user Figure 4C Operation of document entry 421 shown).

[0047] In a possible implementation, the cursor is a text cursor, and the method further includes: the electronic device detects a first input of a user; in response to the first input, the electronic device inputs and displays a fourth text, where the fourth text is generated based on the first input.

[0048] In the embodiment of the present application, when the cursor is a pointer cursor, the electronic device can determine that the user intends to input an operation instruction, and when the cursor is a text cursor, the electronic device can determine that the user intends to input text.

[0049] The first input may be an input performed by the user through voice, and the fourth text may be a text generated by the user through the first input.

[0050] In this way, through the intelligent interaction function, the electronic device can directly record the text input by the user through voice (for example Figure 5E and Fig. 5F As shown in the figure, the text is directly inserted into the email that the user is editing), which is convenient and fast and improves the efficiency of text input.

[0051] In one possible implementation, the intelligent interactive portal is used to display a first menu option, which is generated based on a first interactive object. The method also includes: the electronic device detects an eighth operation of the user on the first menu option; in response to the eighth operation, the electronic device performs an operation corresponding to the first menu option on the first interactive object.

[0052] The first menu option may be an intelligent menu option associated with the content of the first interactive object, and the eighth operation may be, for example, a click operation.

[0053] In the embodiment of the present application, the intelligent interactive portal can not only display the operation instructions input by the user, but also be used to display intelligent menu options.

[0054] In this way, the electronic device can identify the content of the interactive object through the intelligent interaction function, and dynamically generate and display intelligent menu options based on the content, so as to further complete quick operations on the interactive object, which is convenient and efficient, and solves the problem of fixed and single menu functions provided by current applications.

[0055] In one possible implementation, the first interactive object includes information related to the date, and the first menu option is used to add the information related to the date to a schedule reminder application. The electronic device performs an operation corresponding to the first menu option for the first interactive object, specifically including: the electronic device adds the information related to the date to the schedule reminder application.

[0056] The schedule reminder application may be, for example, a calendar application, and the first menu option may be, for example, an “add to calendar” menu option.

[0057] In this way, the user does not need to manually open the schedule reminder application to add the schedule information such as date included in the first interactive object. It can be completed by clicking the first menu option (such as the "Add to Calendar" menu option). It is convenient, fast and efficient, simplifies user operations and improves user experience.

[0058] In a possible implementation, the first interactive object includes a QR code, the first menu option is used to identify the QR code, and the electronic device performs an operation corresponding to the first menu option on the first interactive object, specifically including: the electronic device identifies the QR code.

[0059] In this way, users do not need to manually search for the function option of identifying the QR code on the electronic device. They can complete it by clicking the first menu option (such as the menu option of "Identify QR Code", which is convenient, fast and efficient, simplifies user operations and improves user experience.

[0060] In a possible implementation, the electronic device performs the corresponding operation on the first interactive object by calling a function provided by an application program where the first interactive object is located.

[0061] In an embodiment of the present application, the electronic device can call the function of the original menu option in the application where the interactive object is located. There is no need to jump through multiple levels of menus. The task can be completed by entering only one command, which simplifies user operations, improves human-computer interaction efficiency, and improves user experience.

[0062] In a second aspect, an embodiment of the present application provides an electronic device, which includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method in any possible implementation of the above-mentioned first aspect.

[0063] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are executed on an electronic device, the electronic device executes a method in any possible implementation of the first aspect above.

[0064] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1A-1E This is a schematic diagram of a user interface for performing an erasing operation on image content provided by an embodiment of the present application;

[0066] Figure 1F-1G This is a schematic diagram of a user interface for playing a video provided in an embodiment of the present application;

[0067] Figure 2A-2E It is a schematic diagram of a user interface for performing a document content summary operation provided in an embodiment of the present application;

[0068] Figure 3A-3D This is a schematic diagram of a user interface for performing picture insertion operations provided in an embodiment of the present application;

[0069] Figure 4A-4D is a schematic diagram of a user interface for performing a document search operation provided in an embodiment of the present application;

[0070] Figure 5A-5F A user interface diagram for performing text insertion operations provided in an embodiment of the present application is shown;

[0071] Figure 5G-5J A user interface diagram for performing text translation operations provided in an embodiment of the present application;

[0072] Figure 6A-6C It is a user interface schematic diagram for performing quick operations on interactive objects based on a group of intelligent menu options provided in an embodiment of the present application;

[0073] Figure 6D-Figure 6E This is a schematic diagram of a user interface for performing question-and-answer query operations provided in an embodiment of the present application;

[0074] Fig. 6F is a user interface schematic diagram showing a menu bar provided in an embodiment of the present application;

[0075] Figure 7A-7D It is a schematic diagram of a user interface for saving a group of pictures provided in an embodiment of the present application;

[0076] Figure 8A-8D It is a schematic diagram of a user interface for performing parameter adjustment operations on a picture provided in an embodiment of the present application;

[0077] Fig. 9 This is a flow chart of the interactive method provided by the embodiment of the present application in an intelligent dialog application scenario;

[0078] Fig.10 This is a flow chart of the interactive method provided in the embodiment of the present application in the application scenario of a smart menu;

[0079] Fig.11 is a schematic diagram of a software architecture of an electronic device 100 provided in an embodiment of the present application;

[0080] Fig.12 It is a structural diagram of an electronic device 100 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0082] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0083] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0084] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0085] With the development of science and technology, the way of human-computer interaction has also changed. It has become possible to transform from machine language interaction to natural language interaction. Machine language interaction is mainly completed by clicking on interface controls or physical buttons through keyboard and mouse or touch. Sometimes a task requires multiple levels of jumps to complete, which is cumbersome. Natural language interaction is mainly completed through the user's natural language input (such as language input or text input) combined with powerful natural language processing capabilities. It can avoid level jumps and go straight to the point in one step. Therefore, compared with machine language interaction, natural language interaction can bring users a more convenient and intelligent experience.

[0086] Currently, more and more electronic devices can provide natural language interaction functions, which can support electronic devices to receive language / text instructions from users through natural language input (such as voice input or text input), and can support electronic devices to understand and execute the instructions.

[0087] However, in some scenarios where an interactive object needs to be processed (for example, erasing part of the content in a picture), the user is sometimes unable to accurately express the interactive object through voice / text alone, and the electronic device is unable to accurately and quickly recognize the user's intention. Multiple rounds of natural language interaction may be required to complete the task, resulting in low interaction efficiency and poor user experience.

[0088] In addition, for personal computers (PCs), when users currently perform tasks with many operating steps on PCs, the cursor needs to make a lot of meaningless movements on the screen, and the moving distance is also long, which is complicated and cumbersome, and the user experience is poor.

[0089] For example, the natural language interaction entrance on the PC is usually displayed at a fixed position on the screen. If a user wants to operate an interactive object through the natural language interaction function, the user needs to first select the interactive object and move the cursor to the natural language interaction entrance to perform the corresponding operation. After the operation is completed, the cursor is moved back to the original position. Such frequent movement of the cursor cannot bring an intelligent experience to the user, and the task execution efficiency is also low.

[0090] Based on the above problems, the embodiment of the present application provides an interaction method, which can be applied to an electronic device 100 that can provide a natural language interaction function. Taking the electronic device 100 as a PC as an example, the electronic device 100 can establish a connection with a device with cursor attributes (such as a mouse, a touchpad, etc.). After the connection is established, the electronic device 100 can support the user to start the natural language interaction function of the electronic device 100 (which can be considered as a kind of intelligent interaction function) by performing specific operations on the device with cursor attributes (such as a mouse, a touchpad, etc.) (such as a long press operation on the left and / or right buttons on the mouse, etc.), and display a natural language interaction entrance, wherein the natural language interaction entrance can be formed and displayed by the cursor, or it can be independently displayed near the location of the cursor. Further, the electronic device 100 can detect the voice / text command input by the user through natural language (such as voice input or text input), and display the command at the above-mentioned natural language interaction entrance. After the command input is completed, the electronic device 100 can substitute the interaction object into the input command for understanding and execution, thereby completing the task specified by the user. The above-mentioned interactive object can be determined by the electronic device 100 extracting and identifying the interface content at the cursor position or at the cursor position and its vicinity. In this way, the natural language interactive function can be triggered at the cursor position or its vicinity, so that the operation corresponding to the natural language instruction can be further performed based on the interactive object, and the task can be completed without moving the cursor over a long distance to a fixed natural language interactive entrance, which is convenient and fast, and can solve the problem of inconvenience in expressing interactive objects in natural language.

[0091] It should be noted that, for the convenience of understanding this application, most embodiments of this application are described by taking the natural language interaction portal involved in the prior art as an example. The intelligent interaction function or intelligent interaction portal may also include other types of interaction portals other than natural language, as well as other types of intelligent interaction portals brought about by technological development, which are not limited in this application.

[0092] In addition, by implementing the interaction method provided in the embodiment of the present application, the electronic device 100 can also intelligently identify the content of the interactive object and provide intelligent menu options based on the content, so as to further complete the quick operation of the interactive object. In this way, the problem that the current menu functions are relatively fixed and single, and can only be set based on the type of interactive object can be solved; the electronic device 100 can also call the function of the corresponding menu option in the original context or application through natural language instructions input by the user (such as voice / text instructions). In this way, the problem that complex functions require users to complete by clicking on multi-level menus can be solved. There is no need to jump to different levels, and one step is directly accessed, which simplifies user operations, improves human-computer interaction efficiency, and improves user experience.

[0093] It should be noted that the present embodiment does not limit the specific type of the electronic device 100. For example, the electronic device 100 may be equipped with Or portable electronic devices with other operating systems, such as laptop computers, handheld computers, etc.; for example, the electronic device 100 can also be a non-portable electronic device such as a laptop computer with a touch-sensitive surface or touch panel, a desktop computer with a touch-sensitive surface or touch panel, etc.

[0094] In the embodiment of the present application, taking the electronic device 100 as a PC as an example, the electronic device 100 can establish a connection with a device with cursor attributes (such as a mouse, a touchpad, etc.) in a wired or wireless manner.

[0095] In an embodiment of the present application, taking establishing a connection between the electronic device 100 and the mouse as an example, after the connection is established, the user can control the electronic device 100 through the mouse. For example, the user can trigger the electronic device 100 to start the natural language interaction function through specific operations on the mouse.

[0096] Among them, the above-mentioned specific operations on the mouse may include but are not limited to any of the following operations: a long press operation on the left button on the mouse, a long press operation on the right button on the mouse, a long press operation on the left and right buttons on the mouse, a long press operation or a click operation on other physical buttons on the mouse except the left and right buttons, etc.

[0097] The interaction method provided in the embodiment of the present application can be applied in a variety of scenarios. The following introduces the interaction method provided in the embodiment of the present application in combination with the user interface and several typical application scenarios.

[0098] It should be noted that the following application scenarios are only a few typical application scenarios. The interaction method provided in the embodiments of the present application can also be applied to many other scenarios, and the embodiments of the present application are not limited to this.

[0099] 1. Application scenarios of smart dialog boxes

[0100] Currently, in terms of role positioning, the cursor is only used to indicate the location where the user operation occurs, and in terms of interaction capabilities, the cursor can only support user tapping operations.

[0101] By implementing the interaction method provided in the embodiment of the present application, the cursor can play a greater role and provide users with more new intelligent experiences.

[0102] For example, in terms of character positioning, the cursor can not only be used to indicate where the user action occurs, but the cursor can also be a dialog box based on the interactive object.

[0103] For another example, in terms of interactive capabilities, the cursor not only supports users to perform tapping operations, but also supports users to input operating instructions through voice or directly input text through voice, realizing multiple interactive capabilities such as the cursor can be clicked, spoken, and directly input.

[0104] In the embodiment of the present application, the "smart dialog application scenario" may refer to an application scenario in which the electronic device 100 can provide the cursor with the capability of a smart dialog box by utilizing the interaction method provided in the embodiment of the present application, and support interaction with the user through natural language interaction functions, thereby enabling convenient and quick application scenarios in which corresponding operations can be performed on interactive objects (such as pictures, documents, etc.).

[0105] 1. Perform corresponding operations on the picture

[0106] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to conveniently and quickly perform a variety of operations on the picture, for example, share, collect, copy and other operations on the picture; for example, edit the picture (such as content erasing operation, content replacement operation, content insertion operation, content recognition operation, content beautification operation, style transfer operation, parameter adjustment operation, etc.); etc., the embodiment of the present application does not limit the specific types of the above operations. It is easy to understand that for different types of operations, only the natural language instructions input by the user are different, and the specific implementation process is similar.

[0107] The following takes the operation of erasing the image content as an example to illustrate the specific process of performing the corresponding operation on the image.

[0108] Figure 1A-1E A set of user interfaces for performing an erasing operation on image content is exemplarily shown.

[0109] Take a picture in the Notes app as an example. Figure 1A , Figure 1AThe exemplary user interface 110 may be a user interface of a memo application. It can be seen that the user interface 110 includes a note created by a user (eg, a note named “Rainbow Town Travel Notes”), which includes a picture 111 .

[0110] In the embodiment of the present application, assuming that the electronic device 100 has established a connection with the mouse, then, Figure 1A The pointer cursor 112 exemplarily shown in FIG. 1 is the cursor of a mouse.

[0111] It should be noted that the embodiment of the present application does not limit the display style of the pointer cursor 112. For example, its display style can be Figure 1A The "water drop" style shown in .

[0112] It is easy to understand that the pointer cursor 112 can respond to mouse input (such as click, long press, etc.) to select, move, and other operations on the interface elements displayed in the electronic device 100. Different from the text cursor (also called the insertion symbol), the text cursor can respond to keyboard input to trigger the electronic device 100 to input text in the text input interface. The text cursor can be used to prompt the user to input text, and can also be used to indicate the insertion position of text during the text input process.

[0113] In an embodiment of the present application, a user can trigger the electronic device 100 to select an interactive object through specific operations on the mouse, and can also trigger the electronic device 100 to start a natural language interaction function, wherein the above-mentioned interactive object can be an interactive object subsequently used for natural language interaction, and the above-mentioned interactive object can be determined by the electronic device 100 extracting and identifying the interface content at the cursor position or at the cursor position and its vicinity.

[0114] The embodiment of the present application does not limit the time sequence in which the electronic device 100 selects an interactive object and starts a natural language interaction function.

[0115] The following is an introduction using an example in which the electronic device 100 first selects an interactive object and then starts the natural language interaction function.

[0116] Continue reading Figure 1AIt can be seen that the position of the pointer cursor 112 is the position of the left flower 111a in the picture 111. For example, the electronic device 100 can detect a specific operation of the user on the mouse. In response to the specific operation, the electronic device 100 can extract and identify the interface content at the position of the pointer cursor 112 or at the position of the pointer cursor 112 and its vicinity, so as to determine the interaction object. For example, the interaction object may be the left flower 111a in the picture 111. Then, the left flower 111a in the picture 111 is the interaction object used for subsequent natural language interaction.

[0117] Further, see Figure 1B In response to the above-mentioned user's specific operation on the mouse, the electronic device 100 can start the natural language interaction function. After starting the natural language interaction function, the electronic device 100 can display a natural language interaction entrance. The natural language interaction entrance can be used to prompt the user that the electronic device 100 has started the natural language interaction function, and can also be used for subsequent natural language interaction. Among them, the natural language interaction entrance displayed by the electronic device 100 can be Figure 1B The natural language interaction portal 113 is shown as an example.

[0118] After the natural language interaction function is activated, the electronic device 100 can detect the natural language command input by the user. In the embodiment of the present application, the natural language command input by the user can be a voice command or a text command. The voice command can be input by the user through the microphone on the electronic device 100, and the text command can be input by the user through the keyboard connected to the electronic device 100.

[0119] Continue reading Figure 1B Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to erase the interactive object 111a on the left side of the picture 111 in the picture 111, the electronic device 100 can detect the relevant voice instruction input by the user (for example, voice "erase this") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 113b "erase this").

[0120] Optionally, after the electronic device 100 starts to detect the voice command input by the user, a logo 113a may be displayed on the natural language interaction portal 113, and the logo 113a may be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0121] In some embodiments, if the above-mentioned specific operation of the user on the mouse is a long press operation, the electronic device 100 can start to detect the natural language instructions input by the user in real time after starting the natural language interaction function, regardless of whether the user cancels the long press operation.

[0122] In some embodiments, if the above-mentioned specific operation of the user on the mouse is a long press operation, after the electronic device 100 starts to detect the voice command input by the user, if the electronic device 100 detects that the user has not canceled the long press operation, then the electronic device 100 can continue to detect the voice command input by the user; if the electronic device 100 detects that the user has canceled the long press operation, then the electronic device 100 can no longer detect the voice command input by the user (that is, end collecting voice input).

[0123] In some embodiments, if the above-mentioned specific operation of the user on the mouse is a long press operation, after the electronic device 100 detects the voice command input by the user, regardless of whether the user cancels the long press operation, if the electronic device 100 does not detect that the user continues to input the voice command within a preset period of time, then the electronic device 100 can confirm that the voice command input is complete. In other embodiments, after the electronic device 100 detects the voice command input by the user, if the electronic device 100 detects that the user cancels the long press operation, then the electronic device 100 can confirm that the voice command input is complete.

[0124] Further, see Figure 1B After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to erase the interactive object, the flower 111a on the left side of the picture 111, in the picture 111.

[0125] Furthermore, the electronic device 100 can use a related algorithm to erase the interactive object, the flower 111a on the left side of the picture 111, from the picture 111. After erasing, refer to Figure 1C , the electronic device 100 may no longer display the interactive object of the flower 111a in the picture 111. So far, the picture content erasing operation is completed.

[0126] Continue reading Figure 1B , Figure 1B The exemplary natural language interaction entry 113 may be Figure 1A The pointer cursor 112 is deformed as shown in the exemplary embodiment. After the deformation, the electronic device 100 may only display the natural language interaction entrance 113 (for example, display the natural language interaction entrance 113 at the position before the pointer cursor 112 is deformed), and no longer display the pointer cursor 112. In some embodiments, refer to Figure 1D, the natural language interaction entrance 113 may not be formed by the pointer cursor 112. In this case, the electronic device 100 may display the natural language interaction entrance 113 and the pointer cursor 112 separately, and the pointer cursor 112 is no longer hidden. Preferably, the natural language interaction entrance 113 may be independently displayed near the location of the pointer cursor 112.

[0127] In some embodiments, see Figure 1E After confirming that the user's natural language instruction has been input, the electronic device 100 can Figure 1B The exemplary logo 113a is replaced by Figure 1E The logo 113c shown as an example may be used to indicate that the user's natural language instruction input is complete.

[0128] It should be noted that the voice instruction "erase this" for erasing the image content is merely exemplary and should not be construed as limiting the present application. It is easy to understand that the voice instruction for erasing the image content needs to meet certain conditions, under which the electronic device 100 can determine the position of the object in the image that the user wants to erase. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the erasing object and the erasing action (such as "erase", etc.).

[0129] It should be noted that the embodiment of the present application does not limit the display style of the natural language interaction entrance; the embodiment of the present application does not limit the display position of the natural language interaction entrance, for example, it can be the position of the pointer cursor 112; for another example, it can be a position near the position of the pointer cursor 112; etc. Preferably, the natural language interaction entrance can be formed by the deformation of the pointer cursor 112, and after the deformation, the natural language interaction entrance can be displayed at the position of the pointer cursor 112 before the deformation, so that the process of the pointer cursor 112 being transformed into the natural language interaction entrance can provide users with a more flexible and vivid visual experience.

[0130] It should be noted that the meaning of "natural language interaction portal" has been recorded in the embodiments of the present application, and its name does not constitute any limitation to the embodiments of the present application. In other embodiments of the present application, the natural language interaction portal can be considered as a kind of intelligent interaction portal, and can also be called a natural language (or voice / text) recognition box / prompt box / recognition bar / prompt bar / recognition window / prompt window, etc.

[0131] It should be noted that in the interaction method provided in the embodiment of the present application, the operation of the user selecting an interaction object and triggering the electronic device 100 to start the natural language interaction function can be the same operation, and the specific type of the operation is not limited in the embodiment of the present application.

[0132] It should be noted that in the interaction method provided in the embodiment of the present application, whether the user wants to trigger the electronic device 100 to start the natural language interaction function at the same time after selecting the interaction object can be set by the user himself or by the system default setting of the electronic device 100, and the embodiment of the present application does not limit this.

[0133] It is easy to understand that the interactive method provided in the embodiment of the present application is not limited to the above-mentioned corresponding operations on pictures. It can also support corresponding operations on videos, for example, sharing, collecting, playing and other operations on videos; for example, editing operations on videos (such as specific segment cropping operations, specific screen cropping operations, etc.); and so on. The specific process is similar to the above-mentioned process of performing corresponding operations on pictures, and will not be repeated here.

[0134] In the interface of a video playback application, multiple videos to be played can usually be displayed. Through the interactive method provided in the embodiment of the present application, when a video is in a focused state, the electronic device 100 can support the user to input natural language commands to perform corresponding operations on the video in the focused state (such as play, share, collect, edit, etc.), which is convenient and fast, and improves the interaction efficiency and user experience.

[0135] For example, see Figure 1F , Figure 1F The exemplary user interface may include options corresponding to multiple videos to be played (e.g., option 121 corresponding to video 1, option 122 corresponding to video 2, and option 123 corresponding to video 3). The user may trigger the electronic device 100 to put the corresponding video in a focused state by moving the pointer cursor 112 to the area where the corresponding option is located. For example, Figure 1F As shown, the pointer cursor 112 is in the area where the option 121 is located, and the video 1 is in the focus state. In this case, the electronic device 100 can support the user to play the video by clicking. In the embodiment of the present application, the electronic device 100 can also support the user to input natural language instructions by triggering the natural language interaction function to perform corresponding operations on the video 1 in the focus state. For example, the user can input Figure 1G The exemplary natural language instruction (e.g., "play this video") is used to play video 1; for another example, the user can also start playing video 1 at a certain playback progress by inputting a natural language instruction; etc. The process of the user triggering the natural language interaction function can refer to the content of the aforementioned embodiment, which will not be repeated here.

[0136] 2. Perform corresponding operations on the document

[0137] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to conveniently and quickly perform a variety of operations on the document, for example, sharing, collecting, content overview, searching, opening, specific format conversion and other operations on the document; for example, editing operations on the document (such as content abbreviation operations, picture insertion operations, text insertion operations, etc.); etc., the embodiment of the present application does not limit the specific types of the above operations. It is easy to understand that for different types of operations, only the natural language instructions input by the user are different, and the specific implementation process is similar.

[0138] The following uses the content summary operation, image insertion operation, search operation, open operation, and text insertion operation as examples to illustrate the specific process of performing corresponding operations on the document.

[0139] (1) An overview of the document content (e.g., abbreviations of text paragraphs, etc.)

[0140] Figure 2A-2E A set of user interfaces for performing document content summarization operations is exemplarily shown.

[0141] For example, take a message in the Mail application. Figure 2A , Figure 2A The exemplary user interface 210 may be a user interface of an email application. It can be seen that the user interface 210 includes an email that the user is editing (eg, an email with the subject “Work Meeting Minutes”).

[0142] In the embodiment of the present application, it is also assumed that the electronic device 100 has established a connection with the mouse, then, Figure 2A The pointer cursor 112 shown in the figure is the cursor of the mouse. The introduction of the pointer cursor 112 can refer to the above-mentioned related content, which will not be repeated here.

[0143] In the embodiment of the present application, the user can select the target text (ie, the interactive object) by performing specific operations on the mouse, and can also trigger the electronic device 100 to start the natural language interaction function.

[0144] Continue reading Figure 2A Assuming that the user wants to perform a shorthand operation on a text paragraph 211 (i.e., the target text) in the body of the email, the electronic device 100 can detect that the user performs a drag operation on the mouse. In response to the drag operation, the electronic device 100 can change the pointer cursor 112 from Figure 2A Move to the position shown Figure 2BThe position of the text paragraph 211 shown (for example, the beginning of the text paragraph 211), further, the electronic device 100 can detect the user's specific operation of the mouse, and in response to the specific operation, the electronic device 100 can select the text paragraph 211 and use the text paragraph 211 as an interactive object for subsequent natural language interaction.

[0145] It should be noted that the range of text selected by the electronic device 100 through the above-mentioned specific operation can be set according to a preset rule. For example, the text selected by the electronic device 100 can be only the text at the location where the above-mentioned specific operation is performed, or it can be a larger range of text than the location where the above-mentioned specific operation is performed. The embodiment of the present application does not limit this. The embodiment of the present application only takes the case where the range of text selected by the electronic device 100 through the above-mentioned specific operation is the entire text paragraph (for example, text paragraph 211) as an example.

[0146] Further, see Figure 2C In response to the above-mentioned user's specific operation on the mouse, the electronic device 100 can start the natural language interaction function. After starting the natural language interaction function, the electronic device 100 can display the natural language interaction entrance 212. The natural language interaction entrance 212 can be used to prompt the user that the electronic device 100 has started the natural language interaction function, and can also be used for subsequent natural language interaction.

[0147] Optionally, after the electronic device 100 starts to detect the voice command input by the user, a logo 212a may be displayed on the natural language interaction portal 212, and the logo 212a may be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0148] Preferably, the natural language interaction entrance 212 may be formed by the deformation of the pointer cursor 112 . After the deformation, the natural language interaction entrance 212 may be displayed at the position where the pointer cursor 112 was before the deformation.

[0149] Among them, the timing when the electronic device 100 starts to detect the natural language instructions input by the user and the timing when it ends to detect the natural language instructions input by the user can refer to the relevant content in the aforementioned corresponding operations on the picture, which will not be repeated here.

[0150] Continue reading Figure 2C Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to abbreviate the text paragraph 211 in the body of the email, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "briefly summarize these contents"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 212b "briefly summarize these contents").

[0151] Further, see Figure 1B After the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is to succinctly summarize the selected text (ie, the text paragraph 211).

[0152] Furthermore, the electronic device 100 can simplify and summarize the selected text (ie, the text paragraph 211) through a relevant algorithm. After the simplification and summarization is completed, refer to Figure 2D , the electronic device 100 may display a window 220, which may include a simplified and summarized text 221 (e.g., "XXXXXX"), and may also include options 222 and 223, wherein option 222 may be used to prompt the user to cancel displaying the window 220 and not replace the selected text paragraph 211 with the simplified and summarized text 221, and option 223 may be used to prompt the user to replace the selected text paragraph 211 with the simplified and summarized text 221. If the electronic device 100 detects that the user chooses to replace the text paragraph 211 with the simplified and summarized text 221 through a voice command or a click operation (e.g., a click operation on option 223 by a mouse), then, refer to Figure 2E , the electronic device 100 can Figure 2A The text paragraph 211 shown in FIG. 1 is replaced with the simplified and summarized text 221 and displayed. At this point, the text paragraph abbreviation operation is completed.

[0153] In some embodiments, after the electronic device 100 simplifies and summarizes the selected text (ie, the text paragraph 211), the text paragraph 211 may be directly replaced with the simplistic and summarized text without displaying the text. Figure 2D Window 220 is shown.

[0154] In some embodiments, continue to refer to Figure 2D After confirming that the user's natural language instruction has been input, the electronic device 100 can Figure 2C The exemplary logo 212a is replaced by Figure 2D As shown in the exemplary form, the mark 212c may be used to indicate that the user's natural language instruction input is complete.

[0155] In some embodiments, the operation of triggering the electronic device 100 to select text may not be the above-mentioned specific operation of the user on the mouse, for example, it may be an operation of long pressing the left button of the mouse and dragging the mouse. Figure 2B, the electronic device 100 can detect the operation of the user pressing the left mouse button and dragging the mouse so that the pointer cursor 112 moves from the beginning position of the text paragraph 211 to the end position of the text paragraph 211. In response to this operation, the electronic device 100 can select the text paragraph 211. It is easy to understand that in this case, after selecting the text paragraph 211, the user still needs to trigger the electronic device to start the natural language interaction function through a specific operation on the mouse.

[0156] It should be noted that the above-mentioned voice instruction "briefly summarize these contents" for performing the document content summary operation is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the above-mentioned voice instruction for performing the document content summary operation needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to summarize the document content. Among them, the above-mentioned satisfying "certain conditions" may mean that the voice instruction includes certain keywords for indicating the document content summary action (such as "simplify", "summarize", "overview", "brief description", "abbreviation", etc.).

[0157] (2) Image Insertion

[0158] Figure 3A-3D A set of user interfaces for performing picture insertion operations is exemplarily shown.

[0159] For example, take a note in the Notes app. Figure 3A , Figure 3A The exemplary user interface 310 may be a user interface of a memo application. It can be seen that the user interface 310 includes a note created by the user (eg, a note named “I found a kitten on the roadside”).

[0160] In the embodiment of the present application, it is also assumed that the electronic device 100 has established a connection with the mouse, then, Figure 3A The pointer cursor 112 shown in the figure is the cursor of the mouse. The introduction of the pointer cursor 112 can refer to the above-mentioned related content, which will not be repeated here.

[0161] In the embodiment of the present application, the user can trigger the electronic device 100 to start the natural language interaction function by performing specific operations on the mouse.

[0162] Continue reading Figure 3A Assuming that the user wants to insert a picture at the location of the pointer cursor 112, the electronic device 100 can detect the user's specific operation on the mouse. In response to the specific operation, the electronic device 100 can start the natural language interaction function. After starting the natural language interaction function, refer to Figure 3BThe electronic device 100 may display a natural language interaction entry 311, which may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0163] Optionally, after the electronic device 100 starts to detect the voice command input by the user, a logo 311a may be displayed on the natural language interaction entrance 311, and the logo 311a may be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0164] Preferably, the natural language interaction entrance 311 may be formed by the deformation of the pointer cursor 112 , and after the deformation, the natural language interaction entrance 311 may be displayed at the position where the pointer cursor 112 was before the deformation.

[0165] Among them, the timing when the electronic device 100 starts to detect the natural language instructions input by the user and the timing when it ends to detect the natural language instructions input by the user can refer to the relevant content in the aforementioned corresponding operations on the picture, which will not be repeated here.

[0166] Continue reading Figure 3B , taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to obtain pictures related to cats, so that the obtained pictures related to cats can be inserted into Figure 3A At the position of the pointer cursor 112 shown, the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "Find the cat photos taken by XX device"), and can display the detected relevant voice instructions input by the user in the form of text (for example, text 311b "Find the cat photos taken by XX device").

[0167] Further, see Figure 3B After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to search for cat photos taken by XX device (such as a mobile phone or other device).

[0168] It is easy to understand that the above-mentioned XX device can be a device that establishes a communication connection with the electronic device 100, and the electronic device 100 can obtain the content (such as pictures, videos, documents, etc.) stored on the XX device based on the communication connection.

[0169] Further, the electronic device 100 can obtain cat-related pictures from the XX device (for example, obtain the stored cat pictures from the gallery application of the XX device). After the acquisition is completed, refer to Figure 3C, the electronic device 100 may display a window 320, which may include cat-related pictures (eg, picture 321, picture 322) on the XX device that the electronic device 100 has acquired.

[0170] Further, see Figure 3C , suppose the user wants to Figure 3A The position of the pointer cursor 112 shown in FIG. 1 is where the above-obtained picture, such as picture 321, is inserted. Then, the electronic device 100 can detect the user's operation (such as a click operation) on the picture 321, and in response to the operation, refer to FIG. Figure 3D , the electronic device 100 can be Figure 3A The picture 321 is inserted and displayed at the position of the pointer cursor 112. At this point, the picture insertion operation is completed.

[0171] In some embodiments, when the electronic device 100 obtains multiple pictures, the electronic device 100 can also support the user to select pictures in batches. Furthermore, the electronic device 100 can insert the multiple pictures selected by the user in batches into the document, which is convenient and fast and simplifies user operations.

[0172] In some embodiments, continue to refer to Figure 3C After confirming that the user's natural language instruction has been input, the electronic device 100 can Figure 3B The exemplary logo 311a is replaced by Figure 3C As shown in the exemplary form, the mark 311c can be used to indicate that the user's natural language instruction input is complete.

[0173] Not limited to the above-mentioned picture insertion operation, by implementing the interactive method provided in the embodiment of the present application, other types of content (such as video, etc.) can also be inserted into the document.

[0174] Not limited to the above-mentioned acquisition of inserted content across devices, the electronic device 100 may also acquire inserted content locally or in the cloud based on natural language instructions input by the user.

[0175] It should be noted that the above-mentioned voice instruction for performing content acquisition operations "Find cat photos taken by XX device" is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the above-mentioned voice instruction for performing content acquisition operations needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to acquire content. Among them, the above-mentioned "certain conditions" may mean that the voice instruction includes certain keywords for indicating the content to be acquired (such as "cat photos") and content acquisition actions (such as "acquire", "find", "search", "find", "query", etc.).

[0176] (3) Document search and opening

[0177] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can search for the corresponding document based on the natural language instruction input by the user (for example, search for the corresponding document locally), and after searching for the document, it can also directly open the document. In this way, when the user needs to view a certain document, the user does not need to go to the storage path corresponding to the document to find the document, which is convenient, fast and efficient, simplifies user operations, and improves user experience.

[0178] Figure 4A-4D A set of user interfaces for performing document search operations is exemplarily shown.

[0179] See also Figure 4A , Figure 4A The user interface 410 exemplarily shown may be a desktop of the electronic device 100 .

[0180] In the embodiment of the present application, it is also assumed that the electronic device 100 has established a connection with the mouse, then, Figure 4A The pointer cursor 112 shown in the figure is the cursor of the mouse. The introduction of the pointer cursor 112 can refer to the above-mentioned related content, which will not be repeated here.

[0181] In the embodiment of the present application, the user can trigger the electronic device 100 to start the natural language interaction function by performing specific operations on the mouse.

[0182] Continue reading Figure 4A , assuming that the user wants to search for and open a document, the electronic device 100 can detect a specific operation of the user on the mouse, and in response to the specific operation, the electronic device 100 can start the natural language interaction function. After starting the natural language interaction function, refer to Figure 4B The electronic device 100 may display a natural language interaction entry 411, which may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0183] Optionally, after the electronic device 100 starts to detect the voice command input by the user, a logo 411a may be displayed on the natural language interaction entrance 411, and the logo 411a may be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0184] Preferably, the natural language interaction entrance 411 may be formed by the deformation of the pointer cursor 112 , and after the deformation, the natural language interaction entrance 411 may be displayed at the position where the pointer cursor 112 was before the deformation.

[0185] Among them, the timing when the electronic device 100 starts to detect the natural language instructions input by the user and the timing when it ends to detect the natural language instructions input by the user can refer to the relevant content in the aforementioned corresponding operations on the picture, which will not be repeated here.

[0186] Continue reading Figure 4B Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to open the PowerPoint (PPT) document edited yesterday, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "open the PPT document I edited yesterday"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 411b "open the PPT document I edited yesterday").

[0187] Further, see Figure 4B After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to open the PPT document edited by the user yesterday.

[0188] Furthermore, the electronic device 100 can search locally for all PPT documents edited by the user yesterday. After the search is completed, refer to Figure 4C , the electronic device 100 can display window 420, which can include document entries corresponding to all PPT documents edited by the user yesterday that the electronic device 100 has searched locally (for example, document entry 421, document entry 422). It can be seen that document entry 421 is the document entry corresponding to document 1, and document entry 422 is the document entry corresponding to document 2, wherein each document entry may include relevant information of the document (for example, document icon, document name, last modification time of the document, etc.).

[0189] Further, see Figure 4C Assuming that the user wants to open one of the documents, such as document 1, the electronic device 100 may detect the user's operation (such as a click operation) on the document entry 421, and in response to the operation, refer to Figure 4D , the electronic device 100 can open the document 1 and display the user interface corresponding to the document 1. So far, the document search and opening operation is completed.

[0190] In some embodiments, the user can trigger the electronic device 100 to open the local searched Figure 4C One of the documents shown.

[0191] In some embodiments, when the electronic device 100 searches for only one document, the electronic device 100 may not display the document. Figure 4CInstead of the exemplary window 420 shown, the document can be directly opened and the user interface corresponding to the document can be displayed.

[0192] In some embodiments, continue to refer to Figure 4C After confirming that the user's natural language instruction has been input, the electronic device 100 can Figure 4B The exemplary logo 411a is replaced by Figure 4C The logo 411c is shown as an example. The logo 411c can be used to indicate that the user's natural language instruction input is completed.

[0193] It is easy to understand that the above only takes the document search and opening operation on the desktop of the electronic device 100 as an example, and is not limited to the desktop of the electronic device 100 (the natural language interaction function can be triggered at any position on the desktop). The interaction method provided in the embodiment of the present application can also support document search and opening operations on other user interfaces of the electronic device 100 (the natural language interaction function can be triggered at any position on other user interfaces). It is easy to understand that in the above document search scenario, there may be no identifiable interaction object at the location where the natural language interaction function is triggered.

[0194] It is easy to understand that the above only takes the document search operation as an example and is not limited to the document search operation. The interaction method provided in the embodiment of the present application can also support users to perform other search operations (such as picture search operations, video search operations, folder search operations, etc.).

[0195] It should be noted that the above-mentioned voice instruction for searching and opening documents, "Open the PPT document I edited yesterday", is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the above-mentioned voice instruction for content acquisition operations needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to search and open documents. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating documents to be searched or opened, search or opening actions (such as "search", "open", "find", "query", etc.).

[0196] (4) Text insertion

[0197] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can intelligently identify whether the user intends to input an operation instruction or to input text. After the identification is completed, the electronic device 100 can perform corresponding operations (such as responding to operation instructions, inserting text) based on the user's input, which is convenient, fast and efficient, simplifies user operations, and improves user experience.

[0198] In the embodiment of the present application, when the cursor is a pointer cursor, the electronic device 100 can determine that the user intends to input an operation instruction, and when the cursor is a text cursor, the electronic device 100 can determine that the user intends to input text.

[0199] Figure 5A-5F A set of user interfaces for performing text insertion operations is exemplarily shown.

[0200] For example, take a message in the Mail application. Figure 5A , Figure 5A The exemplary user interface 510 may be a user interface of a mail application. It can be seen that the user interface 510 includes an mail that the user is editing (eg, an mail with the subject "Reply: Work Summary").

[0201] In the embodiment of the present application, it is also assumed that the electronic device 100 has established a connection with the mouse, then, Figure 5A The pointer cursor 112 shown in the figure is the cursor of the mouse. The introduction of the pointer cursor 112 can refer to the above-mentioned related content, which will not be repeated here.

[0202] In the embodiment of the present application, the user can trigger the electronic device 100 to start the natural language interaction function by performing specific operations on the mouse.

[0203] Continue reading Figure 5A Assuming that the user wants to enter text in the body editing area of ​​the email, the electronic device 100 can detect that the user performs a drag operation on the mouse. In response to the drag operation, the electronic device 100 can move the pointer cursor 112 from Figure 5A Move to the position shown Figure 5B The cursor 112 may be changed to a certain position in the body editing area of ​​the mail shown in FIG. Figure 5B The text cursor 511 is shown as an example for display.

[0204] It should be noted that the embodiment of the present application does not limit the display style of the text cursor 511. For example, its display style can be Figure 5B The "vertical line" style shown in the figure can also be an "I" shaped style.

[0205] Furthermore, the electronic device 100 can detect a specific operation of the user on the mouse. In response to the specific operation, the electronic device 100 can start a natural language interaction function. Since the cursor currently displayed by the electronic device 100 is a text cursor, the electronic device 100 can determine that the user intends to insert text in the body editing area of ​​the email.

[0206] Further, see Figure 5CTaking the case of a user inputting text by voice as an example, the electronic device 100 can detect the text input by the user by voice. In the process of the user inputting text by voice, refer to FIG. 5D to FIG. 5E , the electronic device 100 can insert and display the text (such as the text "Received.") in the body editing area of ​​the email in real time based on the user's voice input, until the user's voice input is completed, the electronic device 100 will insert and display all the text input by the user through voice in the body editing area of ​​the email. At this point, the text insertion operation is completed.

[0207] Among them, the timing when the electronic device 100 starts detecting the user's voice input and the timing when it ends detecting the user's voice input can refer to the relevant content in the aforementioned corresponding operations on the picture, which will not be repeated here.

[0208] Optionally, continue to Figure 5C After the electronic device 100 starts the natural language interaction function, it can Figure 5B The exemplary text cursor 511 is transformed into Figure 5C The logo 512 is shown as an example, and the logo 512 can be used to prompt the user to input text by voice.

[0209] Optionally, continue to Figure 5C After the electronic device 100 starts the natural language interaction function, it can display a mark 513, which can be used to remind the user that the electronic device 100 is detecting the user's voice input. Figure 5D and Figure 5E The display position of the logo 513 can also be moved as the text is continuously inserted.

[0210] See also Fig. 5F After the above-mentioned voice input is completed, the electronic device 100 can cancel the display of the mark 513, and can also change the mark 512 back to the original text cursor 511.

[0211] Through the interactive method provided in the embodiment of the present application, the electronic device 100 can also support the user to perform corresponding operations on the above-mentioned inserted text, which is explained below using the translation operation as an example.

[0212] Figure 5G-5J A set of user interfaces for performing text translation operations is exemplarily shown.

[0213] See also Figure 5GAfter selecting the text to be translated (e.g., "Received."), the electronic device 100 can change the text cursor to a pointer cursor 112 when the cursor is located in the area where the selected text is located (the cursor can still be displayed as a text cursor when it is located in an editing area outside the area where the selected text is located). Further, the electronic device 100 can detect a specific operation of the user on the mouse, and in response to the specific operation, the electronic device 100 can start the natural language interaction function. Since the cursor currently displayed by the electronic device 100 is a pointer cursor, the electronic device 100 can determine that the user intends to input an operation instruction.

[0214] Further, see Figure 5H After starting the natural language interaction function, the electronic device 100 may display a natural language interaction entrance 514. The natural language interaction entrance 514 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0215] Optionally, after the electronic device 100 starts to detect the voice command input by the user, a logo 514a may be displayed on the natural language interaction portal 514, and the logo 514a may be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0216] Preferably, the natural language interaction entrance 514 may be formed by the deformation of the pointer cursor 112 , and after the deformation, the natural language interaction entrance 514 may be displayed at the position where the pointer cursor 112 was before the deformation.

[0217] Among them, the timing when the electronic device 100 starts to detect the natural language instructions input by the user and the timing when it ends to detect the natural language instructions input by the user can refer to the relevant content in the aforementioned corresponding operations on the picture, which will not be repeated here.

[0218] Continue reading Figure 5H Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to translate the above-mentioned selected text (for example, "Received."), then the electronic device 100 can detect the relevant voice instruction input by the user (for example, voice "translate into English"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 514b "translate into English").

[0219] Further, see Figure 5H After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to translate the selected text (eg, "Received.") into English.

[0220] Furthermore, the electronic device 100 can translate the selected text into English. After the translation is completed, refer to Fig.5I , the electronic device 100 may display a window 520, which may include a translated text 521 (e.g., "Received."), and may also include options 522 and 523, wherein option 522 may be used to prompt the user to cancel the display of window 520 and not replace the selected text (e.g., "Received.") with the translated text 521, and option 523 may be used to prompt the user to replace the selected text with the translated text 521. If the electronic device 100 detects that the user chooses to replace the selected text with the translated text 521 through a voice command or a click operation (e.g., a click operation on option 523 with a mouse), then, refer to Figure 5J , the electronic device 100 can replace the selected text with the translated text 521 and display it. At this point, the text translation operation is completed.

[0221] In some embodiments, after translating the selected text (eg, "Received."), the electronic device 100 may directly replace the selected text with the translated text 521 (eg, "Received.") without displaying the selected text. Fig.5I Window 520 is shown as an example.

[0222] In some embodiments, continue to refer to Fig.5I After confirming that the user's natural language instruction has been input, the electronic device 100 can Figure 5H The exemplary logo 514a is replaced by Fig.5I As shown in the exemplary form, the mark 514c may be used to indicate that the user's natural language instruction input is complete.

[0223] It should be noted that the voice instruction "Translate into English" for performing the text translation operation is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for performing the text translation operation needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to perform a text translation operation. Among them, the above-mentioned "certain conditions" may mean that the voice instruction includes certain keywords for indicating that the user wants to perform a text translation operation (for example, the name of the language to be translated / converted into, "translation", "conversion" and other keywords).

[0224] It should be noted that the above-mentioned corresponding operations on the document are only illustrated by taking the five operations of document content overview, image insertion, document search, opening, and text insertion as examples. It is not limited to these five operations and can also be more operations. The embodiments of the present application do not limit this.

[0225] It should be noted that the above only uses the corresponding operations on pictures and documents as an example to illustrate the interaction method provided in the embodiment of the present application, and is not limited to this. By implementing the interaction method provided in the embodiment of the present application, more interactive objects (such as more other interactive interface elements) can also be operated, and the embodiment of the present application does not limit this.

[0226] 2. Application scenarios of smart menu

[0227] Currently, menu functions are relatively fixed and single, and can only be set based on the type of interactive object. Moreover, for complex functions, users are required to complete them by clicking on multi-level menus, which is cumbersome.

[0228] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can provide the user with a more intelligent menu function.

[0229] For example, the electronic device 100 can intelligently identify the content of the interactive object, and dynamically generate and display intelligent menu options based on the content, so as to further complete quick operations on the interactive object, which is convenient and efficient.

[0230] For another example, the electronic device 100 can also call the function of the corresponding menu option in the original context or application through natural language commands input by the user (such as voice / text commands). There is no need to jump through multiple levels of menus. Only one natural language command needs to be entered to complete the task, which simplifies user operations, improves human-computer interaction efficiency, and improves user experience.

[0231] In the embodiment of the present application, "smart menu application scenario" may refer to an application scenario in which the electronic device 100 can provide the user with a more intelligent menu function by utilizing the interaction method provided in the embodiment of the present application, thereby being able to conveniently and quickly perform corresponding operations on interactive objects (such as text, pictures, etc.).

[0232] 1. Intelligently identify the content of the interactive object, and dynamically generate and display intelligent menu options based on the content

[0233] Figure 6A-6C A set of user interfaces for performing quick operations on interactive objects based on intelligent menu options is exemplified.

[0234] Take a social application as an example, see Fig. 6A , Fig. 6A The user interface 610 shown as an example may be a chat interface of the social application, and the chat interface may include one or more chat records (eg, a chat record of text 611 in a chat group named "XX Department").

[0235] In the embodiment of the present application, it is also assumed that the electronic device 100 has established a connection with the mouse, then, Fig. 6A The pointer cursor 112 shown in the figure is the cursor of the mouse. The introduction of the pointer cursor 112 can refer to the above-mentioned related content, which will not be repeated here.

[0236] In an embodiment of the present application, the user can trigger the electronic device 100 to recognize the content of the interactive object through specific operations on the mouse, and can also trigger the electronic device 100 to start the natural language interaction function to generate and display intelligent menu options based on the content of the recognized interactive object.

[0237] Continue reading Fig. 6A Assuming that the user wants to perform a corresponding operation on the text 611, the user can first move the pointer cursor 112 to the position where the text 611 is located (for example, Fig. 6A The position of the pointer cursor 112 is shown in the example), further, the electronic device 100 can detect the user's specific operation of the mouse, and in response to the specific operation, the electronic device 100 can recognize the content of the text 611 (ie, the interactive object) and can start the natural language interaction function.

[0238] See also Figure 6B After starting the natural language interaction function, the electronic device 100 may display a natural language interaction portal 612, on which one or more menu options (e.g., menu option 612a) may be displayed. It is easy to understand that the one or more menu options may be generated by the electronic device 100 based on the content of the recognized text 611 (i.e., the interaction object), for example, Fig. 6A The content of the text 611 shown is related to the user's schedule, and the content includes information such as date (for example, June 18th, June 19th), activity name (for example, team building at the seaside), location (for example, Shuangyue Bay, Huizhou City, Guangdong Province), etc. Then, the electronic device 100 can recognize that the text 611 includes the above-mentioned content related to the user's schedule. Further, the electronic device 100 can generate corresponding menu options based on the content, such as menu options related to schedule information (such as date, activity name, location, etc.) (for example, menu option 612a can be the menu option "Add to calendar").

[0239] Continue reading Figure 6B Assuming that the user wants to add the schedule information in the text 611 to the calendar, the electronic device 100 may detect the user's operation (eg, a click operation) on the menu option 612a through the mouse, and in response to the operation, refer to Figure 6C, the electronic device 100 can add the schedule information (such as the name, time, and location of the activity) in the text 611 to the calendar, and display a window 614, which can include the schedule information added to the calendar, thereby prompting the user that the electronic device 100 has successfully added the schedule information in the text 611 to the calendar. In this way, the user does not need to open the calendar application to add the schedule information in the text 611, which is convenient, fast, and efficient, simplifies user operations, and improves user experience.

[0240] Continue reading Figure 6B Optionally, a logo 612b may be displayed on the natural language interaction entrance 612, and the logo 612b may be used to prompt the user that the electronic device 100 has started the natural language interaction function.

[0241] Continue reading Figure 6B Optionally, a prompt 612c is displayed on the natural language interaction entrance 612, and the prompt 612c can be used to prompt the user to input a natural language instruction related to the interaction object. For example, the interaction object is text 611, and the electronic device 100 can recognize that the text 611 includes geographic location information. The prompt 612c can be a prompt for prompting the user to input a natural language instruction related to the geographic location (such as "Huizhou" in the text 611) (for example, "You can say what delicious food is there in Huizhou").

[0242] Optionally, the prompt 612c may also be output in the form of voice broadcast.

[0243] Continue reading Figure 6B Optionally, the electronic device 100 may also display a menu bar 613. It is easy to understand that the menu bar 613 is the original menu bar provided by the above-mentioned social application for chat records. The menu bar 613 may include one or more options (such as a "copy" option, a "reference" option, a "forward" option, a "multi-select" option, a "translate to" option, etc.), and the one or more options may be used to perform corresponding operations on the text 611.

[0244] Continue reading Figure 6C The electronic device 100 can support the user to interact by inputting natural language instructions (such as question and answer queries, etc.), wherein the natural language instructions input by the user can be voice instructions or text instructions. The voice instructions can be input by the user through the microphone on the electronic device 100, and the text instructions can be input by the user through the keyboard connected to the electronic device 100.

[0245] The embodiment of the present application does not limit the triggering method of the electronic device 100 to detect whether the user inputs a voice command or a text command.

[0246] In some embodiments, the triggering method for the electronic device 100 to detect the user input voice command may be that the electronic device 100 starts the natural language interaction function (that is, after starting the natural language interaction function, the user input voice command begins to be detected). In other embodiments, the triggering method for the electronic device 100 to detect the user input voice command may also be that the electronic device 100 detects that the user uses the mouse to point to Figure 6C The exemplary operation (eg, a click operation) of the mark 612b (or the prompt 612c or any other position on the natural language interaction portal 612 except the position where the menu option 612a is located) is shown.

[0247] In some embodiments, the electronic device 100 may also be triggered by detecting that the user inputs a text instruction when the electronic device 100 detects that the user uses the mouse to click on the Figure 6C The exemplary operation (eg, a click operation) of the mark 612b (or the prompt 612c or any other position on the natural language interaction portal 612 except the position where the menu option 612a is located) is shown.

[0248] For example, see Fig.6D Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "What delicious food is there in Huizhou"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 612d "What delicious food is there in Huizhou").

[0249] Continue reading Fig.6D Optionally, after the electronic device 100 starts detecting the voice command input by the user, Figure 6C The exemplary logo 612b is replaced by Fig.6D The logo 612e is shown as an example. The logo 612e can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0250] Further, see Fig.6D After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to search for food in Huizhou.

[0251] Furthermore, the electronic device 100 can search (e.g., search in the cloud) for food in Huizhou. After the search is completed, the electronic device 100 can display search results 615, which can include information related to food in Huizhou that has been searched by the electronic device 100 (e.g., text information, picture information, etc.). In this way, the user does not need to search for the information he wants to obtain in other applications, which is convenient, fast and efficient, simplifies user operations, and improves user experience.

[0252] In some embodiments, continue to refer to Fig. 6E After confirming that the user's natural language instruction has been input, the electronic device 100 can Fig.6D The exemplary logo 612e is replaced by Fig. 6E As shown in the exemplary form, the mark 612b may be used to indicate that the user's natural language instruction input is complete.

[0253] In some embodiments, see Fig. 6F , the electronic device 100 may also Figure 6B The natural language interaction entrance 612 and the menu bar 613 are displayed as one. That is, in the smart menu application scenario, the electronic device 100 can display the natural language interaction entrance and the original menu bar provided in the application as one.

[0254] It should be noted that the voice command "What delicious food is there in Huizhou" for the above-mentioned search operation is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the voice command for the above-mentioned content search operation needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to search for content. Among them, the above-mentioned "certain conditions" may refer to the voice command including certain keywords for indicating the content to be searched, search actions (such as "food", "delicious", "what", "search", "find", "query", etc.).

[0255] 2. The function of calling the corresponding menu option in the original context or application through natural language instructions input by the user (such as voice / text instructions)

[0256] By implementing the method provided in the embodiment of the present application, for tasks that require multi-level menu jumps to complete, the user only needs to enter a natural language command to complete it, thereby simplifying user operations, improving human-computer interaction efficiency, and enhancing user experience.

[0257] It is understandable that there are many tasks that require multi-level menu jumps to complete, for example, saving an object (such as a picture, video, document, etc.) to a certain storage path; for example, editing an object (such as adjusting parameters of a picture); and so on.

[0258] The following uses two application scenarios, namely saving a picture and adjusting picture parameters, as examples to introduce them.

[0259] It is easy to understand that the interaction method provided in the embodiment of the present application is not limited to the application scenarios of the above two multi-level menu jumps. The interaction method provided in the embodiment of the present application can also be applied to other application scenarios of multi-level menu jumps, and the embodiment of the present application does not limit this.

[0260] (1) Save the image

[0261] Figure 7A-7D A set of user interfaces for saving pictures is exemplarily shown.

[0262] Take a social application as an example, see Fig. 7A , Fig. 7A The user interface 710 shown as an example may be a chat interface of the social application, and the chat interface may include one or more chat records (eg, a chat record including picture 711 in a chat group named "XX Department").

[0263] In the embodiment of the present application, it is also assumed that the electronic device 100 has established a connection with the mouse, then, Fig. 7A The pointer cursor 112 shown in the figure is the cursor of the mouse. The introduction of the pointer cursor 112 can refer to the above-mentioned related content, which will not be repeated here.

[0264] In an embodiment of the present application, the user can trigger the electronic device 100 to recognize the content of the interactive object through specific operations on the mouse, and can also trigger the electronic device 100 to start the natural language interaction function to call the function of the corresponding menu option in the original context or application through the natural language instructions input by the user (such as voice / text instructions), and can also trigger the electronic device 100 to generate and display intelligent menu options based on the content of the recognized interactive object.

[0265] Continue reading Fig. 7A , assuming that the user wants to perform a corresponding operation on the image 711, the user can first move the pointer cursor 112 to the position where the image 711 is located (for example, Fig. 7A The position of the pointer cursor 112 is shown in the example), further, the electronic device 100 can detect the user's specific operation of the mouse, and in response to the specific operation, the electronic device 100 can recognize the content of the picture 711 (ie, the interactive object) and can start the natural language interaction function.

[0266] See also Figure 7B After starting the natural language interaction function, the electronic device 100 can display a natural language interaction entrance 612.

[0267] Optionally, a logo 712a may be displayed on the natural language interaction entrance 612, and the logo 712a may be used to prompt the user that the natural language interaction function of the electronic device 100 has been activated.

[0268] Optionally, a prompt 712b (eg, “You can say save as to “Desktop””) may be displayed on the natural language interaction portal 612, and the prompt 712b may be used to prompt the user to complete a task that requires multiple levels of menu jumps by inputting natural language instructions.

[0269] Optionally, the prompt 712b may also be output in the form of voice broadcast.

[0270] Optionally, one or more menu options (e.g., menu option 712c) may be displayed on the natural language interaction portal 612. It is easy to understand that the one or more menu options may be generated by the electronic device 100 based on the content of the recognized picture 711 (i.e., the interactive object). For example, Fig. 7A The picture 711 shown includes a QR code, so the menu option generated by the electronic device 100 can be a menu option related to the QR code (for example, the menu option 712c can be the menu option of "identify QR code"). The user can trigger the electronic device 100 to identify the QR code in the picture 711 by operating the menu option 712c (for example, clicking).

[0271] Optionally, the electronic device 100 may display a menu bar 713. It is easy to understand that the menu bar 713 is the original menu bar provided by the above-mentioned social application for chat records. The menu bar 713 may include one or more options (such as a "copy" option, a "reference" option, a "forward" option, a "multi-select" option, a "save as" option, etc.), and these one or more options can be used to perform corresponding operations on the picture 711.

[0272] Continue reading Figure 7B , the electronic device 100 can support the user to call the function of the corresponding menu option in the original context or application by inputting a natural language instruction, so as to achieve the purpose that for tasks that require multi-level menu jumps to complete, the user only needs to input a natural language instruction to complete it. Among them, the natural language instruction input by the user can be a voice instruction or a text instruction. Among them, the voice instruction can be input by the user through the microphone on the electronic device 100, and the text instruction can be input by the user through the keyboard connected to the electronic device 100.

[0273] The embodiment of the present application does not limit the triggering method of the electronic device 100 to detect whether the user inputs a voice command or a text command.

[0274] In some embodiments, the triggering method for the electronic device 100 to detect the user input voice command may be that the electronic device 100 starts the natural language interaction function (that is, after starting the natural language interaction function, the voice command input by the user begins to be detected). In other embodiments, the triggering method for the electronic device 100 to detect the user input voice command may also be that the electronic device 100 detects that the user uses the mouse to point to Figure 7B The operation (eg, a click operation) of the exemplary mark 712a (or the prompt 712b or any other position on the natural language interaction portal 712 except the position where the menu option 712c is located) is shown.

[0275] In some embodiments, the electronic device 100 may also be triggered by detecting that the user inputs a text instruction when the electronic device 100 detects that the user uses the mouse to click on the Figure 7B The operation (eg, a click operation) of the exemplary mark 712a (or the prompt 712b or any other position on the natural language interaction portal 712 except the position where the menu option 712c is located) is shown.

[0276] For example, see Figure 7C Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to save picture 711 (i.e., the interactive object) to a certain folder (e.g., a human-computer interaction folder), then the electronic device 100 can detect the relevant voice instruction input by the user (e.g., voice "save to human-computer interaction folder"), and can display the detected relevant voice instruction input by the user in the form of text (e.g., text 712d "save to human-computer interaction folder").

[0277] Continue reading Figure 7C Optionally, after the electronic device 100 starts detecting the voice command input by the user, Figure 7B The exemplary logo 712a is replaced by Figure 7C The logo 712e is shown as an example. The logo 712e can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0278] Further, see Figure 7C After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to save the interactive object, picture 711, into the human-computer interaction folder.

[0279] Further, the electronic device 100 may search the storage path of the folder named "human-computer interaction" locally, and after the search is completed, refer to Fig.7D, the electronic device 100 may display a window 714, which may include one or more storage paths (e.g., storage path 714a, storage path 714b) where the folder named "human-computer interaction" that the electronic device 100 has searched for is located. Further, the electronic device 100 may call the "Save As" menu option function provided by the above-mentioned social application to support the user to choose to save the picture 711 to the human-computer interaction folder corresponding to a certain storage path through voice commands or click operations (e.g., click operations on storage path 714a or storage path 714b with a mouse).

[0280] For example, if the user wants to save the picture 711 to the human-computer interaction folder corresponding to the storage path 714a, the electronic device 100 can detect the click operation on the storage path 714a, and in response to the click operation, the electronic device 100 can directly save the picture 711 to the human-computer interaction folder corresponding to the storage path 714a. At this point, the picture save operation is completed. In this way, the user does not need to first click, for example, Figure 7B The "Save As" option in the menu bar 713 shown in the figure, and then manually searching for the human-computer interaction folder and other multi-step operations to complete the task of saving the picture, is convenient, fast and efficient, simplifies user operations, and improves user experience.

[0281] In some embodiments, when the electronic device 100 has searched for only one storage path for the folder of the above-mentioned "human-computer interaction", the electronic device 100 may not display the folder. Fig.7D Instead of the window 714 shown as an example, the image 711 can be directly saved in the "human-computer interaction" folder corresponding to this storage path by calling the "Save As" menu option function provided by the above-mentioned social application.

[0282] In some embodiments, continue to refer to Fig.7D After confirming that the user's natural language instruction has been input, the electronic device 100 can Figure 7C The exemplary logo 712e is replaced by Fig.7D As shown in the exemplary form, the mark 712a may be used to indicate that the user's natural language instruction input is complete.

[0283] It should be noted that the voice instruction "save to human-computer interaction folder" for the above-mentioned save operation is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for the above-mentioned save operation needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to perform a save operation. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating save actions and storage paths (such as "save", "save as", "folder", "desktop", etc.).

[0284] (2) Adjust the image parameters

[0285] Figure 8A-8D A set of user interfaces for performing parameter adjustment operations on a picture is exemplarily shown.

[0286] Take an image processing application as an example, see Fig. 8A , Fig. 8A The exemplary user interface 810 may be a user interface of the image processing application, and the user interface may include a picture 811 to be processed.

[0287] In the embodiment of the present application, it is also assumed that the electronic device 100 has established a connection with the mouse, then, Fig. 8A The pointer cursor 112 shown in the figure is the cursor of the mouse. The introduction of the pointer cursor 112 can refer to the above-mentioned related content, which will not be repeated here.

[0288] In an embodiment of the present application, the user can trigger the electronic device 100 to recognize the interactive object through specific operations on the mouse, and can also trigger the electronic device 100 to start the natural language interaction function to call the function of the corresponding menu option in the original context or application through natural language instructions input by the user (such as voice / text instructions).

[0289] Continue reading Fig. 8A , assuming that the user wants to perform a corresponding operation on the image 811, the user can first move the pointer cursor 112 to the position where the image 811 is located (for example, Fig. 8A The position of the pointer cursor 112 is shown in the example), further, the electronic device 100 can detect the user's specific operation of the mouse, and in response to the specific operation, the electronic device 100 can recognize the picture 811 (ie, the interactive object) and can start the natural language interaction function.

[0290] See also Figure 8B After starting the natural language interaction function, the electronic device 100 can display a natural language interaction entrance 812.

[0291] Optionally, a logo 812a may be displayed on the natural language interaction entrance 812, and the logo 812a may be used to prompt the user that the electronic device 100 has started the natural language interaction function.

[0292] Optionally, a prompt 812b (eg, "You can say fill this with white") may be displayed on the natural language interaction portal 812, and the prompt 812b may be used to prompt the user to complete a task that requires multiple levels of menu jumps by inputting natural language instructions.

[0293] Optionally, the prompt 812b may also be output in the form of voice broadcast.

[0294] Optionally, the electronic device 100 can display a menu bar 813. It is easy to understand that the menu bar 813 is the original menu bar provided by the above-mentioned image processing application for the image to be processed. The menu bar 813 can include one or more options (such as a "cancel selection" option, a "select inverse selection" option, a "feather" option, a "fill" option, a "special effect" option, etc.), and these one or more options can be used to perform corresponding operations on the image 811.

[0295] Continue reading Figure 8B , the electronic device 100 can support the user to call the function of the corresponding menu option in the original context or application by inputting a natural language instruction, so as to achieve the purpose that for tasks that require multi-level menu jumps to complete, the user only needs to input a natural language instruction to complete it. Among them, the natural language instruction input by the user can be a voice instruction or a text instruction. Among them, the voice instruction can be input by the user through the microphone on the electronic device 100, and the text instruction can be input by the user through the keyboard connected to the electronic device 100.

[0296] The embodiment of the present application does not limit the triggering method of the electronic device 100 to detect whether the user inputs a voice command or a text command.

[0297] In some embodiments, the triggering method for the electronic device 100 to detect the user input voice command may be that the electronic device 100 starts the natural language interaction function (that is, after starting the natural language interaction function, the user input voice command begins to be detected). In other embodiments, the triggering method for the electronic device 100 to detect the user input voice command may also be that the electronic device 100 detects that the user uses the mouse to point to Figure 8B The exemplary operation (eg, a click operation) of the mark 812a (or the prompt 812b or any other position on the natural language interaction portal 812) is shown.

[0298] In some embodiments, the electronic device 100 may also be triggered by detecting that the user inputs a text instruction when the electronic device 100 detects that the user uses the mouse to click on the Figure 8B The exemplary operation (eg, a click operation) of the mark 812a (or the prompt 812b or any other position on the natural language interaction portal 812) is shown.

[0299] For example, see Figure 8C Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to adjust the parameter of the Gaussian blur radius of the image 811, for example, adjusting the value of the Gaussian blur radius to 5, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, voice "Gaussian blur adjusted to 5"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 812d "Gaussian blur adjusted to 5").

[0300] Continue reading Figure 8C Optionally, after the electronic device 100 starts detecting the voice command input by the user, Figure 8B The exemplary logo 812a is replaced by Figure 8C The logo 812e is shown as an example. The logo 812e can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0301] Further, see Figure 8C After the above voice command is input, the electronic device 100 can also analyze the above voice command to determine that the operation corresponding to the above voice command is to adjust the value of the Gaussian blur radius of the image 811 to 5.

[0302] Furthermore, the electronic device 100 can call the function of the menu option "Gaussian blur" provided by the above-mentioned image processing application to Fig. 8A The Gaussian blur radius value of image 811 in the image is automatically adjusted to 5 and can be Fig. 8A The updated display shows the picture 811 after the Gaussian blur radius value is automatically adjusted to 5. At this point, the operation of adjusting the Gaussian blur radius parameter of the picture is completed. In this way, the user does not need to first enter the second-level, third-level, etc. menu bar to find the Gaussian blur menu option, and then click the menu option to enter the Gaussian blur setting interface to complete the task of adjusting the Gaussian blur radius parameter in multiple steps. This is convenient, fast and efficient, simplifies user operations, and improves user experience.

[0303] Optionally, continue to Fig.8D In order to give users more room for independent selection, the electronic device 100 Fig. 8ABefore updating and displaying the picture 811 after the Gaussian blur radius value is automatically adjusted to 5, the electronic device 100 may also display a window 814, which may be used by the user to preview the picture 811 after the Gaussian blur radius value is automatically adjusted to 5, and may also be used by the user to manually adjust the Gaussian blur radius value of the picture 811 again. The window 814 may also include a "Confirm" option and a "Cancel" option. If the electronic device 100 detects that the user operates the "Confirm" option, then the electronic device 100 may Fig. 8A The image 811 after the Gaussian blur radius value is automatically adjusted to 5 is updated and displayed; if the electronic device 100 detects that the user operates the "Cancel" option, then the electronic device 100 does not adjust the Gaussian blur radius value of the image 811, and does not update the display. Fig. 8A Picture 811 in the gallery.

[0304] In some embodiments, continue to refer to Fig.8D After confirming that the user's natural language instruction has been input, the electronic device 100 can Figure 8C The exemplary logo 812e is replaced by Fig.8D As shown in the exemplary form, the mark 812a may be used to indicate that the user's natural language instruction input is complete.

[0305] It should be noted that the above-mentioned voice instruction for adjusting the image parameters "Gaussian blur adjusted to 5" is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the above-mentioned voice instruction for adjusting the image parameters needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to adjust the image parameters. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the parameter type to be adjusted, the parameter adjustment action, and the value corresponding to the parameter type (such as "Gaussian blur", "adjustment", etc.).

[0306] Fig. 9 A specific process of the interaction method provided in an embodiment of the present application in the above-mentioned smart dialog application scenario is exemplified.

[0307] like Fig. 9 As shown, the method can be applied to the electronic device 100. The specific steps of the method are described in detail below:

[0308] S901. The electronic device 100 detects a specific operation of a user on a device with cursor attributes (such as a mouse, a touchpad, etc.). In response to the specific operation, the electronic device 100 starts a natural language interaction function.

[0309] Specifically, taking the electronic device 100 as a PC as an example, the electronic device 100 can establish a connection with a device with cursor attributes (such as a mouse, touchpad, etc.) via a wired or wireless manner. Assuming that the electronic device 100 establishes a connection with the mouse, after the connection is established, the electronic device 100 can detect the user's specific operation on the mouse. In response to the specific operation, the electronic device 100 can start the natural language interaction function.

[0310] Among them, the above-mentioned specific operations on the mouse may include but are not limited to any of the following operations: a long press operation on the left button on the mouse, a long press operation on the right button on the mouse, a long press operation on the left and right buttons on the mouse, a long press operation or a click operation on other physical buttons on the mouse except the left and right buttons, etc.

[0311] After starting the natural language interaction function, the electronic device 100 can display a natural language interaction entrance so that the electronic device 100 can subsequently detect the natural language instructions (such as voice instructions or text instructions) input by the user through the natural language interaction function, and can display the natural language instructions at the natural language interaction entrance.

[0312] Among them, after the electronic device 100 establishes a connection with the mouse, the mouse cursor can be displayed. The above-mentioned natural language interaction entrance can be formed by the deformation of the cursor and displayed at the position where the cursor was before the deformation, or it can not be formed by the deformation of the cursor and be displayed independently at a position near the cursor position.

[0313] The above-mentioned specific operation may also trigger the electronic device 100 to select an interactive object, so that the electronic device 100 may subsequently perform corresponding operations on the interactive object based on the above-mentioned input natural language instructions.

[0314] Among them, the specific process of triggering the electronic device 100 to start the natural language interaction function and display the natural language interaction entrance and select the interaction object through the above-mentioned specific operation can refer to the relevant text description in the aforementioned user interface embodiment, which will not be repeated here.

[0315] S902: The electronic device 100 detects that a user inputs a natural language instruction, and determines an interactive object based on the natural language instruction and the cursor position.

[0316] Specifically, after starting the natural language interaction function, the electronic device 100 can use the natural language interaction function to detect the natural language instructions input by the user. After the natural language instructions are input, the electronic device 100 can determine whether there is a default object (that is, the natural language instruction does not include the interactive object) or a pronoun (that is, the natural language instruction includes words such as this, it, etc. used to refer to the interactive object). If so (that is, the natural language instruction does not include keywords used to indicate the interactive object), the electronic device 100 can extract and identify the interactive object at the above-mentioned cursor position (that is, the position of the cursor when the user performs the above-mentioned specific operation on the mouse) or the interface content at the above-mentioned cursor position and its vicinity, thereby determining the interactive object. If not (that is, the natural language instruction includes keywords used to indicate the interactive object), the electronic device 100 can execute the following step S903.

[0317] In some embodiments, when there is no default object in the natural language instruction (ie, the natural language instruction includes an interactive object), the electronic device 100 may perform corresponding operations on the interactive object according to the interactive object included in the natural language instruction.

[0318] Exemplarily, assuming that the natural language instruction is "open the PPT document I edited yesterday", it can be seen that the interactive object included in the natural language instruction is "the PPT document edited yesterday", then the electronic device 100 can search locally and open the PPT document edited by the user yesterday.

[0319] In other embodiments, when the interactive object included in the natural language instruction is different from the interactive object selected by triggering the electronic device 100 through the above-mentioned specific operation, the electronic device 100 may perform a corresponding operation on the interactive object according to the interactive object included in the natural language instruction.

[0320] For example, assuming that the natural language instruction is "brighten the picture", it can be seen that the interactive object included in the natural language instruction is "picture", and the interactive object selected by the electronic device 100 triggered by the above specific operation is a part of the picture (for example, the above Figure 1A In the example of the flower 111a on the left side of the picture 111, in this case, the electronic device 100 can increase the overall brightness of the picture through a relevant algorithm, rather than just increasing the brightness of a certain part of the picture. That is to say, the interactive object determined by the electronic device 100 is the entire picture mentioned above, rather than a certain part of the picture.

[0321] In other embodiments, when there is a default object or pronoun in the natural language instruction, the electronic device 100 may prioritize the smallest unit object that can be identified at the cursor position (such as a control, a cutout body, a text segmentation, etc.) as the interactive object, and then the electronic device 100 may determine whether to expand the scope of the interactive object based on the natural language instruction. If so, the scope of the interactive object may be expanded to determine a new interactive object, and the operation corresponding to the natural language instruction may be performed on the new interactive object.

[0322] Exemplarily, assuming that the natural language instruction is "erase", it can be seen that there is a default object in the natural language instruction. Then, the electronic device 100 can use the smallest unit object that can be identified at the above cursor position as the interactive object. For example, a part of a picture can be used as the interactive object. Further, the electronic device 100 can erase the part in the picture through a relevant algorithm.

[0323] For example, assuming that the natural language instruction is "erase this", it can be seen that the natural language instruction includes the pronoun "this", then the electronic device 100 can also use the smallest unit object that can be identified at the above cursor position as an interactive object. For example, a part of a picture can be used as an interactive object. Further, the electronic device 100 can erase the part in the picture through a relevant algorithm.

[0324] Exemplarily, assuming that the natural language instruction is "copy this paragraph to the memo", it can be seen that the natural language instruction includes the pronoun "this paragraph". Assuming that the interactive object selected by the electronic device 100 triggered by the above specific operation is a part of the text in a text paragraph (that is, the selected text), then the electronic device 100 can determine the scope of the interactive object to be expanded based on the "this paragraph" in the above natural language instruction. Further, the electronic device 100 can identify the context of the above selected text through relevant algorithms (such as text detection and recognition algorithms such as OCR), thereby obtaining the entire text paragraph where the above selected text is located. Further, the electronic device 100 can determine that the new interactive object is the entire text paragraph where the above selected text is located, and can copy the entire text paragraph where the above selected text is located to the memo.

[0325] S903: The electronic device 100 responds to the natural language instruction based on the interactive object.

[0326] Specifically, after determining the interactive object, the electronic device 100 can analyze and understand the natural language instruction input by the user in combination with the interactive object. After the analysis and understanding, the electronic device 100 can respond to the natural language instruction, that is, perform the operation corresponding to the natural language instruction for the interactive object. For example, perform corresponding operations on pictures (such as sharing, collection, copying, editing, etc.); for another example, perform corresponding operations on documents (such as sharing, collection, content overview, search, opening, editing, etc.); and so on.

[0327] Among them, the specific process of the electronic device 100 responding to the natural language instructions input by the user can refer to the relevant text description in the aforementioned user interface embodiment, which will not be repeated here.

[0328] By implementing the above Fig. 9 In the interactive method provided by the illustrated embodiment, when the electronic device 100 successfully establishes a connection with a device with cursor attributes (such as a mouse, a touchpad, etc.), the electronic device 100 can support the user to start the natural language interactive function of the electronic device 100 by performing specific operations on the device with cursor attributes, and display a natural language interactive entrance at or near the cursor location. Furthermore, the electronic device 100 can detect the voice / text instructions input by the user in natural language, and display the instructions at the above-mentioned natural language interactive entrance. After the instruction is input, the electronic device 100 can substitute the interactive object into the input instruction for understanding and execution. Among them, the above-mentioned interactive object can be determined by the electronic device 100 extracting and identifying the interface content at the cursor location or at and near the cursor location. In this way, the interactive object is more clearly referred to, which can solve the problem of inconvenience in expressing the interactive object in natural language. In addition, the natural language interaction function can be triggered at or near the cursor position, so that the operation corresponding to the natural language instruction can be further performed based on the interactive object. There is no need to move the cursor over a long distance to a fixed natural language interaction entrance, and natural language instructions can be directly input, which is convenient and fast. Moreover, there is no need to add additional physical buttons on the above-mentioned devices with cursor attributes to trigger the natural language interaction function, saving development costs.

[0329] Fig.10 The specific process of the interactive method provided in the embodiment of the present application in the above-mentioned smart menu application scenario is exemplified.

[0330] like Fig.10 As shown, the method can be applied to the electronic device 100. The specific steps of the method are described in detail below:

[0331] S1001. The electronic device 100 detects a specific operation of a user on a device with cursor attributes (such as a mouse, a touchpad, etc.). In response to the specific operation, the electronic device 100 starts a natural language interaction function.

[0332] The specific execution process of step S1001 can refer to the above Fig. 9 The relevant text description about step S901 is not repeated here.

[0333] S1002. The electronic device 100 determines an interactive object based on the position of the cursor, identifies the content of the interactive object, and dynamically generates and displays intelligent menu options based on the content.

[0334] Specifically, in response to the above-mentioned specific operation, the electronic device 100 can extract and identify the interactive object at the position of the above-mentioned cursor (that is, the position of the cursor when the user performs the above-mentioned specific operation) or the interface content at the position of the above-mentioned cursor and its vicinity, so as to determine the interactive object. After determining the interactive object, the electronic device 100 can identify the content of the interactive object based on relevant algorithms (such as computer vision algorithms, service recommendation algorithms, etc.), and dynamically generate and display intelligent menu options based on the content (such as the aforementioned menu options). Figure 6B The menu options 612a shown in the example, Figure 7B Menu option 712c) is shown as an example.

[0335] In the embodiment of the present application, after starting the natural language interaction function, the electronic device 100 may also display a natural language interaction entrance, and the above-mentioned intelligent menu options may be displayed on the natural language interaction entrance.

[0336] S1003, the electronic device 100 detects that the user inputs a natural language instruction, and responds to the natural language instruction based on the interactive object; or, the electronic device 100 detects the user's operation on the intelligent menu option, and responds to the operation.

[0337] In an embodiment of the present application, the electronic device 100 can support the user to perform quick operations on the above-mentioned interactive objects by inputting natural language instructions to call the functions of the corresponding menu options in the original context or application, and can also support the user to perform quick operations on the above-mentioned interactive objects by operating the above-mentioned intelligent menu options (i.e., menu options associated with the content of the above-mentioned interactive objects).

[0338] Exemplarily, the electronic device 100 detects that the user inputs a natural language instruction, and responds to the natural language instruction based on the interactive object. Specifically, the electronic device 100 detects that the natural language instruction input by the user includes a keyword (such as "save as", "save", "Gaussian blur", etc.) used to instruct the electronic device 100 to call the function of the corresponding menu option in the original context or application. In this case, the electronic device 100 can call the function of the menu option indicated by the natural language instruction, and perform corresponding operations on the interactive object based on the function of the menu option (such as save as, parameter adjustment, etc.). In this way, there is no need to jump to multiple levels of menus. The user only needs to enter a natural language instruction to complete the task, which simplifies user operations, improves human-computer interaction efficiency, and improves user experience.

[0339] Exemplarily, the electronic device 100 detects the user's operation on the above-mentioned intelligent menu option and responds to the operation. Specifically, the electronic device 100 detects the user's click operation on the above-mentioned intelligent menu option. In response to the operation, the electronic device 100 can perform a quick operation on the above-mentioned interactive object based on the function of the above-mentioned intelligent menu option. For example, if the content of the above-mentioned interactive object includes content related to the user's schedule, then the above-mentioned intelligent menu option can be, for example, the menu option of "Add to Calendar". In this way, the user does not need to open the calendar application to add the schedule information in the text 611, which is convenient and quick; for another example, if the content of the above-mentioned interactive object includes a QR code, then the above-mentioned intelligent menu option can be, for example, the menu option of "Recognize QR Code". In this way, the user does not need to find the menu option for identifying the QR code provided by the current application or other applications. Clicking the currently displayed "Recognize QR Code" menu option can quickly complete the recognition of the QR code, which is convenient and quick.

[0340] It is easy to understand that the above-mentioned intelligent menu options only take "add to calendar" and "recognize QR code" as examples, and are not limited to this. There may be more other menu options, and the embodiments of the present application are not limited to this.

[0341] The specific execution process of the above step S1003 can refer to the relevant text description in the above user interface embodiment, which will not be repeated here.

[0342] By implementing the above Fig.10The interactive method provided by the illustrated embodiment can support the user to start the natural language interactive function of the electronic device 100 by performing specific operations on the device with cursor attributes when the electronic device 100 successfully establishes a connection with the device with cursor attributes (such as a mouse, a touchpad, etc.). The electronic device 100 can also determine the interactive object based on the position of the cursor, identify the content of the interactive object, and dynamically generate and display intelligent menu options based on the content, so as to further complete the quick operation of the interactive object. In this way, the problem that the current menu function is relatively fixed and single and can only be set based on the type of interactive object can be solved; the electronic device 100 can also call the function of the corresponding menu option in the original context or application through the natural language command (such as voice / text command) input by the user. In this way, the problem that the user needs to complete the complex function by clicking on the multi-level menu can be solved, and the hierarchical jump is eliminated. The user only needs to enter a natural language command to complete it, which simplifies the user operation, improves the efficiency of human-computer interaction, and improves the user experience.

[0343] The following introduces a software structure of an electronic device 100 provided in an embodiment of the present application.

[0344] Fig.11 The software structure of an electronic device 100 provided in an embodiment of the present application is exemplified.

[0345] like Fig.11 As shown, the software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes a layered architecture system as an example to exemplify the software structure of the electronic device 100.

[0346] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system can be divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the system library, and the kernel layer.

[0347] The application layer can include a series of application packages.

[0348] like Fig.11 As shown, the application package may include applications such as camera, gallery, calendar, input method, map, voice assistant, WLAN, Bluetooth, music, video, mailbox, memo (not shown in the figure), etc.

[0349] Among them, the voice assistant can be an application with voice / text command recognition function. For example, after detecting the user's voice input, voice recognition technology, natural language processing technology, etc. can be used to identify the user's intention and respond in a targeted manner (such as voice answering, starting an application, changing device configuration, etc.). Among them, the electronic device 100 can always turn on the voice assistant, or it can start the voice assistant after detecting user operation (such as long pressing an interface element / button operation, etc.), or it can start the voice assistant after detecting a wake-up word, which is not limited here.

[0350] The above-mentioned voice / text instructions may refer to voice / text for controlling the electronic device 100 to perform one or more operations.

[0351] It should be noted that the name of the voice assistant application is merely a term used in the embodiments of the present application. Its meaning has been recorded in the embodiments of the present application, and its name does not constitute any limitation to the embodiments of the present application.

[0352] In an embodiment of the present application, the electronic device 100 can establish a connection with a device with cursor attributes (such as a mouse, a touchpad, etc.). After the connection is established, the electronic device 100 can be triggered to start a natural language interaction function through a specific operation of the user on the device with cursor attributes (such as a long press operation on the left and / or right buttons on the mouse, etc.), so that the electronic device 100 can subsequently detect the natural language instructions (such as voice instructions or text instructions) input by the user through the natural language interaction function. In addition, the above-mentioned specific operation can also trigger the electronic device 100 to select an interactive object, so that the corresponding operation can be performed on the interactive object based on the natural language instructions input by the user. The above-mentioned specific operation can also trigger the electronic device 100 to identify the content of the interactive object, and dynamically generate and display intelligent menu options based on the content, so as to further complete the quick operation on the interactive object. For details, please refer to the relevant text descriptions in the above-mentioned embodiments, which will not be repeated here.

[0353] In an embodiment of the present application, the electronic device 100 can call the function provided by the application program where the interactive object is located through the natural language instructions input by the user, thereby performing corresponding operations on the interactive object.

[0354] In an embodiment of the present application, the natural language interaction function initiated by the electronic device 100 may be a system capability provided by the electronic device 100. In some embodiments, the system capability may be implemented based on a voice assistant application or an input method application.

[0355] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0356] like Fig.11 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, and the like.

[0357] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0358] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, browsing history, and bookmarks.

[0359] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. The display interface can be composed of one or more views. For example, it can include a view for displaying text and a view for displaying images.

[0360] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0361] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0362] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0363] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0364] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0365] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0366] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0367] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0368] A 2D graphics engine is a drawing engine for 2D drawings.

[0369] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, Bluetooth driver, and sensor driver.

[0370] The structure of an electronic device 100 provided in an embodiment of the present application is introduced below.

[0371] Fig.12 The structure of an electronic device 100 provided in an embodiment of the present application is exemplarily shown.

[0372] like Fig.12 As shown, the electronic device 100 may include a processor 110, an internal memory 120, an external memory interface 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a display screen 151, a mouse 152, a camera 153, an audio module 160, a speaker 161, a receiver 162, a microphone 163, an earphone interface 164, a wired communication module 171, a wireless communication module 172, an antenna 1, a sensor module 180, etc. The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a temperature sensor 180C, a touch sensor 180D, an ambient light sensor 180E, etc.

[0373] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, in some embodiments, in addition to the hardware described above, the electronic device 100 may also have a motor, a mobile communication module (2G / 3G / 4G / 5G), a SIM card interface, an eSIM chip, and the like. Therefore, with respect to the specific hardware structure of the electronic device 100, it may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently, depending on the specific circumstances.

[0374] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0375] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0376] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0377] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0378] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0379] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0380] The internal memory 120 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 120. The internal memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100, etc. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0381] The external memory interface 121 can be used to connect an external memory card, such as a mobile hard disk, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 121 to implement a data storage function, such as storing music, video and other files in an external storage hard disk.

[0382] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0383] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 120, the external memory, the display screen 151, the camera 153, and the wireless communication module 172. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0384] The electronic device 100 implements the display function through a GPU, a display screen 151, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 151 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0385] The display screen 151 is used to display images, videos, etc. The display screen 151 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 151, where N is a positive integer greater than 1.

[0386] The mouse 152 can be used to receive user operations to control the electronic device 100 to achieve different functions. For example, the user can trigger the electronic device 100 to select an interactive object by a specific operation on the mouse, and can also trigger the electronic device 100 to start a natural language interaction function, so that the electronic device 100 can further perform corresponding operations on the interactive object based on the natural language interaction function; and so on. It is not limited to the mouse 152, which is a device with cursor attributes, but can also be other devices with cursor attributes (such as a touchpad, etc.) to achieve control of the electronic device 100.

[0387] The electronic device 100 can realize the shooting function through the ISP, the camera 153, the video codec, the GPU, the display screen 151 and the application processor.

[0388] The camera 153 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 153, where N is a positive integer greater than 1.

[0389] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0390] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0391] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0392] The network communication function of the electronic device 100 can be implemented through the wired communication module 171, the wireless communication module 172, the antenna 1, the modem processor and the baseband processor, etc. Through the network communication function of the electronic device 100, the electronic device 100 can communicate with other electronic devices.

[0393] Antenna 1 is used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0394] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 161, a receiver 162, etc.), or displays an image or video through a display screen 151. In some embodiments, the modem processor may be an independent device.

[0395] The wired communication module 171 can provide wired communication solutions including Ethernet, local area network, Internet, etc. applied to the electronic device 100. The wired communication module 171 can be one or more devices integrating at least one communication processing module.

[0396] The wireless communication module 172 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 172 can be one or more devices integrating at least one communication processing module. The wireless communication module 172 receives electromagnetic waves via the antenna 1, modulates the frequency of the electromagnetic wave signal and performs filtering processing, and sends the processed signal to the processor 110. The wireless communication module 172 can also receive the signal to be sent from the processor 110, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 1.

[0397] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 160 , the speaker 161 , the receiver 162 , the microphone 163 , the headphone jack 164 , and the application processor.

[0398] The audio module 160 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 160 can also be used to encode and decode audio signals. In some embodiments, the audio module 160 can be arranged in the processor 110, or some functional modules of the audio module 160 can be arranged in the processor 110.

[0399] The speaker 161 , also called a “speaker”, is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 161 .

[0400] The receiver 162, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 162 close to the human ear.

[0401] Microphone 163, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to the microphone 163 to input the sound signal into the microphone 163. The electronic device 100 can be provided with at least one microphone 163. In other embodiments, the electronic device 100 can be provided with two microphones 163, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 163 to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0402] The earphone interface 164 is used to connect a wired earphone and can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0403] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 151. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 151, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0404] The touch sensor 180D is also called a "touch panel". The touch sensor 180D is used to detect a touch operation applied thereto or thereabout. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. A visual output related to the touch operation can be provided through the display screen 151.

[0405] It should be understood that Fig.12 The electronic device 100 shown is only an example, and the electronic device 100 may have more Fig.12More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Fig.12 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0406] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0407] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0408] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An interaction method, applied to an electronic device, wherein the electronic device includes a screen, characterized in that: The method comprises: The electronic device detects a first operation; In response to the first operation, the electronic device starts an intelligent interaction function and displays an intelligent interaction entrance, wherein the intelligent interaction function is used for the electronic device to interact with the user through voice commands or text commands, wherein the intelligent interaction entrance is displayed at or near a first position, and the first position is a position of a cursor on the screen when the first operation is detected; The electronic device detects that a user inputs a first instruction through the intelligent interaction portal, and determines that the interaction object is a first interaction object based on the first instruction; The electronic device executes the operation corresponding to the first instruction on the first interactive object.

2. The method according to claim 1, characterized in that The electronic device is connected to an accessory device, and the accessory device is used to receive the first operation and the user's operation of moving the cursor.

3. The method according to claim 2, characterized in that The accessory device is a mouse, the mouse includes a left button and a right button, and the first operation is an operation of long pressing the left button, or an operation of long pressing the right button, or an operation of long pressing the left button and the right button.

4. The method according to any one of claims 1 to 3, characterized in that: The electronic device determines, based on the first instruction, that the interaction object is a first interaction object, specifically including: In a case where the first instruction does not include a first keyword, the first keyword is a keyword used to indicate the interactive object, and the electronic device determines that the first interactive object is an object that can be identified at the first position; or, In a case where the first instruction includes the first keyword, the electronic device determines that the interaction object indicated by the first keyword is the first interaction object.

5. The method according to any one of claims 1 to 4, characterized in that: The intelligent interaction entrance is formed by the deformation of the cursor.

6. The method according to any one of claims 1 to 5, characterized in that: After the electronic device starts the intelligent interaction function, the method further includes: The electronic device outputs a first prompt, where the first prompt is used to prompt a user to input a recognizable instruction corresponding to the first interactive object.

7. The method according to any one of claims 1 to 6, characterized in that: After the electronic device detects that the user inputs a first instruction, the method further includes: If the electronic device detects that the user cancels the first operation, the electronic device no longer detects instructions input by the user.

8. The method according to any one of claims 1 to 7, characterized in that: The first instruction does not include a keyword directly indicating a name of an interactive object.

9. The method according to any one of claims 1 to 8, characterized in that: The cursor is a pointer cursor, the first interactive object is a picture or a first content in the picture, the first content is a part of the content in the picture, and the electronic device performs an operation corresponding to the first instruction on the first interactive object, specifically including: In a case where the first instruction is used to indicate an intention to perform a second operation on the first interactive object, the electronic device performs the second operation on the first interactive object, wherein the second operation includes any one of the following operations: sharing, collecting, copying, erasing, replacing, inserting, beautifying, identifying, style transfer, parameter adjustment, and saving.

10. The method according to claim 9, characterized in that In a case where the second operation is a save operation, the electronic device performs the second operation on the first interactive object, specifically including: The electronic device saves the first interactive object to a target location, where the target location is a default storage location, or the target location is determined based on a keyword included in the first instruction for indicating a storage location.

11. The method according to any one of claims 1 to 8, characterized in that: The cursor is a pointer cursor, the first interactive object is a video, and the electronic device performs an operation corresponding to the first instruction on the first interactive object, specifically including: In a case where the first instruction is used to indicate an intention to perform a third operation on the video, the electronic device performs the third operation on the video, wherein the third operation includes any one of the following operations: sharing, collecting, playing, cropping, and saving.

12. The method according to any one of claims 1 to 8, characterized in that: The cursor is a pointer cursor, the first interactive object is a first text, and the electronic device performs an operation corresponding to the first instruction on the first interactive object, specifically including: In the case where the first instruction is used to indicate an intention to summarize the first text, the electronic device summarizes the first text to generate a second text, and displays a first window, wherein the first window includes the second text; or, In the case where the first instruction is used to indicate an intention to translate the first text, the electronic device translates the first text to generate a third text, and displays a second window, wherein the second window includes the third text; or, In the case where the first instruction is used to indicate an intention to query the first text, the electronic device queries the first text and displays a query result, wherein the query result includes relevant information of the first text.

13. The method according to claim 12, characterized in that The method further comprises: The electronic device detects a fourth operation; in response to the fourth operation, the electronic device replaces the first text with the second text and displays the second text; or, The electronic device detects a fifth operation; in response to the fifth operation, the electronic device replaces the first text with the third text and displays the third text.

14. The method according to any one of claims 1 to 8, characterized in that: The cursor is a pointer cursor, and the electronic device performs an operation corresponding to the first instruction on the first interactive object, specifically including: In the case where the first instruction is used to indicate the intention to search for a first object, the electronic device searches for the first object and displays search results, the search results including information of one or more of the first objects, and the type of the first object includes any one of the following: picture, video, document.

15. The method according to claim 14, characterized in that Before the electronic device searches for the first object, the method further includes: The electronic device displays a first user interface, where the first user interface is a user interface corresponding to the first document; The electronic device displays the search results, specifically including: The electronic device displays the search results on the first user interface; After the electronic device displays the search results, the method further includes: The electronic device detects a sixth operation; In response to the sixth operation, the electronic device inserts the first object into the first document.

16. The method according to claim 14, characterized in that The first object is a second document, and before the electronic device searches for the first object, the method further includes: The electronic device displays a desktop; The electronic device displays the search results, specifically including: The electronic device displays the search results on the desktop; After the electronic device displays the search results, the method further includes: The electronic device detects a seventh operation; In response to the seventh operation, the electronic device opens the second document.

17. The method according to any one of claims 1 to 8, characterized in that: The cursor is a text cursor, and the method further comprises: The electronic device detects a first input from a user; In response to the first input, the electronic device inputs and displays a fourth text, where the fourth text is generated based on the first input.

18. The method according to any one of claims 1 to 17, characterized in that: The intelligent interaction portal is used to display a first menu option, where the first menu option is generated based on the first interaction object, and the method further includes: The electronic device detects an eighth operation of the user on the first menu option; In response to the eighth operation, the electronic device performs the operation corresponding to the first menu option on the first interactive object.

19. The method according to claim 18, characterized in that The first interactive object includes information related to a date, the first menu option is used to add the information related to the date to a schedule reminder application, and the electronic device performs an operation corresponding to the first menu option on the first interactive object, specifically including: The electronic device adds the date-related information to the schedule reminder application.

20. The method according to claim 18, characterized in that The first interactive object includes a QR code, the first menu option is used to identify the QR code, and the electronic device performs an operation corresponding to the first menu option on the first interactive object, specifically including: The electronic device recognizes the QR code.

21. The method according to any one of claims 1 to 20, characterized in that: The electronic device performs the corresponding operation on the first interactive object by calling a function provided by an application program where the first interactive object is located.

22. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in any one of claims 1-21.

23. A computer storage medium, characterized in that The computer storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 21.

Images