Interaction method and device, electronic equipment and medium

Abstract

The invention discloses an interaction method and device, electronic equipment and a medium, and belongs to the technical field of electronic equipment. Under the condition that the obtained first data is matched with a first instruction in an instruction set, the first instruction is displayed at a first position of a screen, the instruction set comprises at least one instruction, and the at least one instruction comprises the first instruction; in response to a first operation on the first instruction, a first action associated with the first instruction is executed.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to an interaction method, device, electronic equipment and medium. Background Art

[0002] With the development of electronic device technology, users can experience the functions of electronic devices by interacting with them, and can also interconnect with other devices. At present, the interaction methods supported by electronic devices mainly include touch screen operation, voice assistant, etc. However, traditional touch screen operation and voice assistant require users to find the corresponding function in a complex menu and perform specific touch operations or speak specific voice commands to realize the corresponding function, which is not intuitive and convenient enough. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an interactive method, device, electronic device and medium, which can automatically display instructions based on data obtained by the electronic device. The user only needs to perform a first operation on the displayed instruction to make the electronic device perform the corresponding action, without having to search for the corresponding function in a complex menu, making the interaction simpler and more convenient.

[0004] In a first aspect, an embodiment of the present application provides an interaction method, which is applied to a first electronic device, and the method includes:

[0005] In a case where the acquired first data matches a first instruction in the instruction set, displaying the first instruction at a first position on the screen, wherein the instruction set includes at least one instruction, and the at least one instruction includes the first instruction;

[0006] In response to a first operation on the first instruction, a first action associated with the first instruction is performed.

[0007] In a second aspect, an embodiment of the present application provides an interactive device, applied to a first electronic device, the device comprising:

[0008] A display module, configured to display the first instruction at a first position on the screen when the acquired first data matches a first instruction in the instruction set, wherein the instruction set includes at least one instruction, and the at least one instruction includes the first instruction;

[0009] The processing module is used to execute a first action associated with the first instruction in response to a first operation on the first instruction.

[0010] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0011] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0012] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0013] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0014] In an embodiment of the present application, the electronic device autonomously obtains first data, matches the first data with a pre-edited instruction set, displays a first instruction in the instruction set that matches the first data at a first position on the screen, and executes a first action associated with the first instruction in response to a first operation on the first instruction. According to an embodiment of the present application, the electronic device autonomously obtains data during operation, and determines whether there are matching instructions based on the obtained data, and displays the instructions at the first position of the screen when there are matching instructions, so that the user only needs to perform a first operation on the displayed instructions to make the electronic device perform the corresponding action, without having to search for the corresponding function in a complex menu, making the interaction simpler and more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flowchart of an interaction method provided in some embodiments of the present application;

[0016] Figure 2 is a first position schematic diagram provided in some embodiments of the present application;

[0017] Figure 3 is a flowchart of an interaction method provided by other embodiments of the present application;

[0018] Figure 4 is a schematic diagram of instruction type options provided in some embodiments of the present application;

[0019] Figure 5 is a schematic diagram of an instruction page provided in some embodiments of the present application;

[0020] Figure 6 is a schematic diagram of an instruction page provided in some other embodiments of the present application;

[0021] Figure 7 is a schematic diagram of an instruction page provided in some other embodiments of the present application;

[0022] Figure 8 is a schematic diagram of an agent dialog box provided in some embodiments of the present application;

[0023] Fig. 9 This is a schematic diagram of a chat interface of a social application provided in some embodiments of the present application;

[0024] Fig.10 is a schematic diagram of a device list provided in some embodiments of the present application;

[0025] Fig.11 is a schematic diagram of an agent dialog box provided in some other embodiments of the present application;

[0026] Fig.12 This is a schematic diagram of an instruction confirmation page provided in some embodiments of the present application;

[0027] Fig.13 is a flowchart of an interaction method provided by other embodiments of the present application;

[0028] Fig.14 It is a schematic diagram of folding and displaying instructions provided in some embodiments of the present application;

[0029] Fig.15 is a schematic diagram of an instruction list provided in some embodiments of the present application;

[0030] Fig.16 is a schematic diagram of instruction details provided by some embodiments of the present application;

[0031] Fig.17 is a flowchart of an interaction method provided by other embodiments of the present application;

[0032] Fig.18 is a schematic diagram of a device list provided in some embodiments of the present application;

[0033] Fig.19 is a schematic diagram of the structure of an interactive device provided in some embodiments of the present application;

[0034] Fig. 20 is a block diagram of an electronic device provided by some embodiments of the present application;

[0035] Fig.21 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0037] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0038] The following, in conjunction with the accompanying drawings, describes in detail the interaction method, interaction device, electronic device, storage medium and program product provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0039] The interaction method provided in the present application can be applied to electronic device interaction scenarios, such as scenarios where a user interacts with an electronic device, scenarios where electronic devices interact with each other, etc. Figure 1-Figure 18 The interactive method provided in the embodiment of the present application is described in detail. It should be noted that the interactive method provided in the embodiment of the present application can be executed by an electronic device. In the embodiment of the present application, the interactive method provided in the embodiment of the present application is described by taking the electronic device executing the interactive method as an example.

[0040] See also Figure 1 , is a flow chart of an interactive method provided in an embodiment of the present application, the method is applied to a first electronic device, the first electronic device includes but is not limited to a mobile phone, a car computer, a laptop computer, a tablet computer, a smart wearable device and other electronic devices. For ease of description, the following embodiment of the present application is described by taking the first electronic device as a mobile phone as an example.

[0041] like Figure 1 The method shown includes the following steps 110 to 120, which are described in detail below.

[0042] Step 110: When the acquired first data matches a first instruction in the instruction set, display the first instruction at a first position on the screen.

[0043] In some embodiments of the present application, an instruction set is pre-edited in the first electronic device, the instruction set includes at least one instruction, and the at least one instruction includes the first instruction. The instructions in the instruction set are all effective instructions, and the effective instructions refer to instructions that can be successfully received, understood and executed as expected by the first electronic device, and finally produce corresponding effects.

[0044] In some embodiments of the present application, a data acquisition module is installed in the first electronic device, and the first electronic device can obtain the first data through the data acquisition module during operation. Among them, the data acquisition module includes various sensors and data interfaces. Exemplarily, the data acquisition module includes a microphone, an application program interface, etc. The first data includes but is not limited to audio data collected by a microphone, display content data obtained by application program interfaces such as screenshots and screen recordings, and connection data obtained by a data interface for data interaction with other devices. Exemplarily, the audio data includes the sound of the user speaking collected by the microphone of the first electronic device. The display content data includes the content presented on the screen of the first electronic device obtained by the screenshot software or the screen recording software, including text, images, videos, etc. These contents can be user input, system feedback, or provided by websites, applications, operating systems, etc. For example, the display content includes the chat interface content when the user chats through the social software on the first electronic device. The connection data includes the relevant data of the device connected to the first electronic device and / or the data sent to the first electronic device by the device connected to the first electronic device. Among them, the setting connected to the first electronic device refers to a device that is connected to the first electronic device through a wired connection technology such as Ethernet, optical fiber, or through a wireless connection technology such as Bluetooth, WiFi, ZigBee, and is capable of data interaction with the first electronic device. For example, the device connected to the first electronic device includes a vehicle-mounted computer, a tablet computer, a mobile phone, and other devices that are interconnected with the first electronic device via Bluetooth. The relevant data of the device connected to the first electronic device includes, but is not limited to, the status information and location information of the device connected to the first electronic device, wherein the status information includes, but is not limited to, the power status, processor status, memory status, network status, software status, etc. of the device.

[0045] In some embodiments of the present application, the first electronic device is provided with an intelligent agent, and the first electronic device can be controlled by the intelligent agent to obtain the first data during the operation of the first electronic device when the intelligent agent is turned on. An intelligent agent refers to an agent that can perceive the environment and take actions to achieve a specific goal. It can be software, hardware or a system, and has autonomy, adaptability and interaction capabilities. Exemplarily, the first electronic device is a mobile phone, and the intelligent agent refers to a mobile phone intelligent agent installed in the mobile phone. The mobile phone intelligent agent refers to a multimodal assistant that can actively complete tasks according to the user's intention, and has complete cognitive, decision-making, execution and memory capabilities. It can accurately identify the content in the mobile phone screen interface, automatically operate various applications in the mobile phone, and complete the tasks assigned by the user. The core functions of the mobile phone intelligent agent include perception, decision-making and execution. It can perceive environmental information through data acquisition modules such as sensors, analyze and make decisions using artificial intelligence algorithms, and intervene and control the environment through actuators. The mobile phone intelligent agent can also collaborate with other intelligent agents or humans to complete complex tasks together.

[0046] In some embodiments of the present application, after acquiring the first data, the first electronic device matches the acquired first data with instructions in the instruction set, and if there is an instruction matching the first data in the instruction set, the instruction matching the first data is determined as the first instruction, and the first instruction is displayed at a first position on the screen of the first electronic device. The first position is a screen area on the screen of the first electronic device.

[0047] In some embodiments of the present application, the screen of the first electronic device may be of any screen type, for example, the screen type of the first electronic device may be a special-shaped screen or a full-screen screen, and the special-shaped screen includes but is not limited to a hole-punch screen, a water drop screen, a pill screen, a notch screen, etc. The first position on the screen of the first electronic device may be a screen area specified by the user according to actual needs, or may be a fixed screen area pre-set in the first electronic device.

[0048] In some embodiments of the present application, before the above-mentioned step 110, the user can set the screen area coordinates corresponding to the first position in the first electronic device according to actual needs. In this way, in the above-mentioned step 110, the first electronic device can determine the screen area corresponding to the screen area coordinates set by the user as the first position, and then display the first instruction in the screen area.

[0049] In some other embodiments of the present application, the screen of the first electronic device is a special-shaped screen, and the screen includes a screen area for placing a camera or other sensors, hereinafter referred to as a sensor area, and the first position can be pre-set to a screen area including the sensor area. Figure 2The hole-punch screen shown includes a hole-punch area 210 for installing sensors such as cameras, and the first position can be a screen area 220 that includes the hole-punch area 210. In this way, when the first data acquired by the first electronic device matches the first instruction, the screen area 220 is determined as the first position, and then the first instruction is displayed in the screen area 220. By setting the first position as the screen area including the sensor area, the space around the sensor area is cleverly utilized, and the sensor area that may have been regarded as a screen defect is converted into an interactive display area with practical value, which not only reduces the impact of the hole-punch on the overall visual effect of the screen, but also gives full play to the function of this area, thereby improving the utilization rate of the screen space.

[0050] In some embodiments of the present application, the first electronic device may display the first instruction at the first position through an interactive display control. The interactive display control allows the user to control the displayed content or interact with it through direct operations such as clicking, dragging, zooming, etc.

[0051] Step 120: In response to a first operation on the first instruction, a first action associated with the first instruction is executed.

[0052] In some embodiments of the present application, the first operation is a confirmation operation of the first instruction displayed at the first position, which is used to instruct the first electronic device to perform an action associated with the first instruction. Exemplarily, the first operation includes but is not limited to a click input of the first instruction displayed at the first position by a user through a touch device such as a finger or a stylus, or a voice instruction input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and the embodiments of the present invention are not limited thereto. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a dragging gesture, a pressure recognition gesture, a long-press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long-press input or a short-press input. For example, the above-mentioned first operation can be: a click operation of the user on the first instruction displayed at the first position through a finger. The above-mentioned first operation can also be: a semantic instruction issued by the user to confirm the execution of the first instruction.

[0053] In some embodiments of the present application, each instruction in the instruction set has a pre-set associated action, based on which, in response to a first operation on the first instruction, the first electronic device controls the application program interface, the hardware controller and other related functional modules to execute the first action associated with the first instruction. Exemplarily, each instruction in the instruction set includes an instruction execution condition and an instruction associated action, the first electronic device compares the acquired first data with the instruction execution condition of the instruction in the instruction set, determines the instruction in the instruction set whose instruction execution condition matches the first data as the first instruction, and executes the instruction associated action corresponding to the first instruction when receiving the first operation of the user on the first instruction.

[0054] For example, the instruction set includes a first preset instruction, the instruction execution condition of the first preset instruction is to hear buying buns between 8 and 10 in the morning, the instruction-associated action of the first preset instruction is to open the payment interface of the payment application, and the first data obtained by the first electronic device at 8:30 in the morning includes audio data of "Boss, buy three vegetarian buns". The first electronic device determines that the first data matches the instruction execution condition of the first preset instruction by comparing the first data with the instruction execution conditions of each instruction in the instruction set. At this time, the first electronic device determines the first preset instruction as the first instruction displayed at the first position, and opens the payment interface of the payment application upon receiving the first operation of the user on the first instruction.

[0055] For another example, the instruction set includes a second preset instruction, and the instruction execution condition of the second preset instruction is that "remember" is seen in the chat interface of the social application and the chat interface stays on the screen for a preset period of time. The instruction-associated action of the second preset instruction is to open a calendar recording application. The first data acquired by the first electronic device includes display content data, and the display content data is data in the chat interface of the social application. The data in the chat interface includes "remember", and the chat interface stays on the screen for a period of time greater than the preset period of time. The first electronic device determines that the first data matches the instruction execution condition of the second preset instruction by comparing the first data with the instruction execution conditions of each instruction in the instruction set. At this time, the first electronic device determines the second preset instruction as the first instruction displayed at the first position, and opens the calendar recording application upon receiving the first operation of the user on the first instruction.

[0056] For another example, the instruction set includes a third preset instruction, the instruction execution condition of the third preset instruction is that a first vehicle connected to a first electronic device travels to a highway toll station, the instruction-associated action of the third preset instruction is to open a payment interface of a payment application, and the first data acquired by the first electronic device includes connection data, which indicates that the first vehicle has arrived at the highway toll station. The first electronic device determines that the first data matches the instruction execution condition of the third preset instruction by comparing the first data with the instruction execution conditions of each instruction in the instruction set. At this time, the first electronic device determines the third preset instruction as the first instruction displayed at the first position, and opens the payment interface of the payment application upon receiving a first operation of the user on the first instruction.

[0057] In the interactive method provided by the embodiment of the present application, the electronic device autonomously obtains first data, matches the first data with a pre-edited instruction set, displays a first instruction matching the first data at a first position on the screen, and executes a first action associated with the first instruction in response to a first operation on the first instruction. According to the embodiment of the present application, the electronic device autonomously obtains data during operation, and determines whether there is a matching instruction based on the obtained data. If there is a matching instruction, the instruction is displayed at the first position of the screen, so that the user only needs to perform a first operation on the displayed instruction to make the electronic device perform the corresponding action, without having to search for the corresponding function in a complex menu, making the interaction simpler and more convenient.

[0058] In some embodiments, the instruction set may include multiple types of instructions. Among them, the instruction type can be divided according to the data type of the first data acquired by the first electronic device. For example, the data type of the first data includes audio data, display content and connection data, then the instruction type may include audio type, display type and connection type, wherein the audio type refers to the instruction type corresponding to the audio data, the display type refers to the instruction type corresponding to the display content data, and the connection type refers to the instruction type corresponding to the connection data. Based on this, when editing the instruction set, the instructions of each instruction type can be edited separately.

[0059] In some embodiments, Figure 3 As shown, before the above step 110, the first electronic device can edit the instruction set through the following steps 310-330, which are described in detail below.

[0060] Step 310. In response to a selection operation on a first type in the instruction type options, obtain instruction setting information of the first type, where the first type is any type in the instruction type options.

[0061] In some embodiments of the present application, the instruction type option is an option corresponding to the instruction type and available for user selection, and one instruction type option corresponds to one instruction type. The instruction type may include one or more, and accordingly, the instruction type options corresponding to the instruction type include one or more. In order to facilitate users to more intuitively understand the instruction type corresponding to each instruction type option and to facilitate distinguishing different instruction type options, each instruction type option may include a type identifier of the instruction type corresponding to the instruction type option, wherein the type identifier may include text, images, symbols, etc. For example, Figure 4 As shown, the instruction type options include instruction type option 410, instruction type option 420 and instruction type option 430, wherein the instruction type corresponding to instruction type option 410 is an audio type, and the type identification in instruction type option 410 is "hear", the instruction type corresponding to instruction type option 420 is a display type, and the type identification in instruction type option 420 is "see", and the instruction type corresponding to instruction type option 430 is a connection type, and the type identification in instruction type option 410 is "connect".

[0062] In some embodiments of the present application, in order to reduce the occupation of the screen of the first electronic device, the first electronic device displays the instruction option type when receiving the option display operation. Based on this, before the above step 310, the user can input the option display operation to the first electronic device, and the first electronic device displays the instruction type option in response to the option display operation, and then the user can select the first type from the instruction type options according to actual needs.

[0063] In some embodiments of the present application, the option display operation is an operation for instructing the first electronic device to display an instruction type option. The option display operation includes but is not limited to a click input of the user to the first position through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which can be determined according to actual usage requirements and is not limited in the embodiments of the present application. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, the screen of the first electronic device is as follows: Figure 2 As described above, a camera is provided at the hole-punch area 210 , based on which, the option display operation may be: a short press operation, a long press operation, etc. of the hole-punch area 210 by the user through a finger.

[0064] In some embodiments of the present application, the first type is any instruction type selected by the user from the instruction type options according to actual needs. The first type is usually the instruction type corresponding to the instruction that the user needs to edit.

[0065] In some embodiments of the present application, a selection operation is used to select the type of instruction that the user wants to edit. The user can perform a selection operation on any type of instruction type according to actual needs. The first electronic device, in response to the user's selection operation on the instruction type option, determines the instruction type corresponding to the instruction type option selected by the selection operation as the first type that the user wants to edit. Exemplarily, the selection operation may be but is not limited to the user's click input of the instruction type to be edited through a touch device such as a finger or a stylus, or a voice instruction input by the user, or a specific gesture input by the user, or other feasible input. The specific operation can be determined according to actual usage needs and is not limited in the embodiments of the present application. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc. It can also be a long press input or a short press input. For example, Figure 4 As shown, if the instruction type corresponding to the instruction that the user wants to edit is an audio type, the user can click on the instruction type option 410, and the first electronic device determines the audio type corresponding to the instruction type option 410 as the first type selected by the user.

[0066] In some embodiments of the present application, the first electronic device determines that the user needs to edit the first type of instruction in response to the user's selection operation on the first type. At this time, the first electronic device can obtain instruction setting information corresponding to the instruction to be edited, which is input by the user to the first electronic device. The instruction setting information is information used to generate a new instruction.

[0067] In some embodiments of the present application, it is considered that the editing of instructions includes but is not limited to viewing, adding, deleting, modifying and other editing, and the instruction setting information needs to be obtained only when a new instruction is needed. Therefore, the first electronic device can respond to the selection operation of the first type, and before obtaining the instruction setting information, it can first display the instruction page corresponding to the first type. The instruction page corresponding to the first type is a page for editing the instructions of the first type. Different instruction types have different corresponding instruction pages, but each instruction page includes the first control and the instructions of the corresponding instruction type stored in the instruction set. Among them, the first control is a control for indicating a new instruction.

[0068] Exemplarily, the first type is an audio type, and the instruction page of the first type is as follows: Figure 5The page shown includes a first control 510 and an audio instruction 520 stored in the instruction set. The first control 510 is a control for indicating the addition of a new audio instruction, and the audio instruction is an audio type instruction.

[0069] Exemplarily, the first type is a display type, and the instruction page of the first type is as follows: Figure 6 The page shown includes a first control 610 and a display instruction 620 stored in an instruction set. The first control 610 is a control for indicating a newly added display instruction, and the display instruction is a display type instruction.

[0070] Exemplarily, the first type is a connection type, and the instruction page of the first type is as follows: Figure 7 The page shown includes a first control 710 and a connection instruction 720 stored in the instruction set. The first control 710 is a control for indicating a newly added connection instruction, and the connection instruction is a connection type instruction.

[0071] The user can determine whether to add a new instruction of the first type based on the instructions displayed in the instruction page. If the user determines that a new instruction of the first type needs to be added, the user can perform a third operation on the first control in the instruction page. The first electronic device determines that a new instruction of the first type needs to be added in response to the user's third operation on the first control, and then obtains the first type of instruction setting information.

[0072] In some embodiments of the present application, the third operation is a selection operation on the first control, which is used to indicate the addition of a new first type of instruction. Exemplarily, the third operation includes but is not limited to a click input of the user to the first control through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which can be determined according to actual usage requirements and is not limited in the embodiments of the present invention. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0073] In some embodiments of the present application, the first electronic device can obtain the instruction setting information through its own agent. When setting instructions of different instruction types, different methods can be used to obtain the instruction setting information.

[0074] In some embodiments of the present application, when the first type is an audio type, the method of obtaining instruction setting information may include the following steps:

[0075] Outputting inquiry information for inquiring instruction setting information through the agent dialog box;

[0076] Obtain the answer information entered by the user in the agent dialog box based on the query information;

[0077] The reply information is determined as the first type of instruction setting information.

[0078] For example, Figure 8 As shown, when the first type is the audio type, the first electronic device responds to the user's third operation on the first control, interacts with the user through the intelligent agent dialog box 800, and outputs to the user an inquiry message 810 such as "Can you tell me what to do when I hear what content and at what time?" to guide the user to provide the first type of instruction setting information. After seeing the inquiry message 810, the user can input a reply message 820 such as "Around eight to ten in the morning, when you hear me say buy buns, open the payment interface of the payment application" in the intelligent agent dialog box 800 through input methods such as voice input or text input. At this time, the first electronic device determines the reply message 820 as instruction setting information.

[0079] In some embodiments of the present application, when the first type is a display type, the method of obtaining instruction setting information may include the following steps:

[0080] In response to a user input to a first area of ​​the screen, acquiring display content data of the first area;

[0081] The display content data of the first area is determined as the first type of instruction setting information.

[0082] Among them, the first area is an area on the screen containing the instruction setting information that the user wants. In the case where the first type is a display type, the first electronic device monitors the display content on the screen in response to the user's third operation on the first control. At this time, the user can browse any content in the first electronic device according to actual needs. During the user's browsing process, when the instruction setting information that is desired to be used to set the first type of instruction is found, the screen area where the desired instruction setting information is located is input, and the input is used to instruct the first electronic device to obtain the display content data in the screen area. Exemplarily, the input includes but is not limited to: the user uses a touch device such as a finger or a stylus to circle the screen area containing the instruction setting information that the user wants on the screen, the frame selection operation, or the voice command input by the user, or the specific gesture input by the user, or other feasible input, which can be determined according to actual use needs, and the embodiment of the present invention is not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0083] In response to the input, the first electronic device determines the screen area corresponding to the input as the first area, and obtains the display content data of the first area as the first type of instruction setting information. Fig. 9 As shown, when the user is browsing the chat interface 900 of the social application, he finds that "Remember to buy groceries and go home to cook tomorrow night" is the desired instruction setting information, so he performs a circle selection operation on area 910 where "Remember to buy groceries and go home to cook tomorrow night" is located. In this way, the first electronic device can determine area 910 as the first area selected by the user, thereby obtaining the display content data of the first area "Remember to buy groceries and go home to cook tomorrow night", and determine the obtained display content data as the instruction setting information.

[0084] In some embodiments of the present application, it is considered that there may be the same or similar display content data in different applications, and some display content data may only be related to instructions when they appear in a specific application. For example, the display content data "Remember to buy vegetables and go home to cook tomorrow night" may appear in the chat interface of the social application, or in the reading application or information application, and the display content data only plays a reminder role when it appears in the chat interface of the social application, and when it appears in the reading application or information application, it often has no reminder effect. Therefore, only when the display content data appears in the chat interface of the social application is it related to the instruction related to the reminder, and when it appears in the reading application or information application, it is not related to the reminder. Based on this, in order to make the instructions generated based on the instruction setting information more in line with the user's expectations, when the first type is the display type, when obtaining the instruction setting information, in addition to obtaining the display content data in the first area, the name of the application described in the display content data and other related information can also be obtained, and the relevant information of the application can also be determined as the instruction setting information. In this way, when generating the display type instruction, the factor of the application can be considered, thereby improving the accuracy of the generated instructions.

[0085] In other embodiments of the present application, when the first type is a display type, the instruction setting information can also be obtained in a manner similar to that of obtaining the instruction setting information of the audio type, that is, when the first type is a display type, the first electronic device can also output inquiry information to the user through the intelligent agent dialog box, and then determine the reply information of the user based on the inquiry information as the instruction setting information. For example, when the first type is a display type, the first electronic device outputs inquiry information such as "Can you tell me what to do when what content is displayed in what application" to the user through the intelligent agent dialog box, and the user can input reply information such as "When "Remember" is displayed in a social application, open the schedule application to record the corresponding schedule" in the intelligent agent dialog box based on the inquiry information. In this way, the first electronic device can determine the user's reply information as the instruction setting information.

[0086] In some embodiments of the present application, when the first type is a connection type, a method for obtaining the first type of instruction setting information may include the following steps:

[0087] displaying a device list, the device list including devices connected to the first electronic device;

[0088] In response to a selection operation on a first device in the device list, obtaining a dialog information corresponding to the first device in the agent dialog box, wherein the first device is any device in the device list;

[0089] The dialog information is determined as the first type of instruction setting information.

[0090] Exemplarily, when the first type is a connection type, the first electronic device displays, in response to the user's third operation on the first control, Fig.10 The device list 1000 shown in FIG. 1 includes the identifiers of each device connected to the first electronic device. The identifiers may be text or icons. Fig.10 Taking the representation of the device as an icon as an example, the user can perform a selection operation on the identifier of any device displayed in the device list 1000 according to actual needs. The first electronic device responds to the selection operation input by the user in the device list 1000, determines the device corresponding to the device identifier selected by the user as the first device, and then outputs the query information related to the first device through the intelligent agent dialog box. The user replies with the reply information related to the first device based on the query information. At this time, the first electronic device can determine the reply information related to the first device in the intelligent agent dialog box as the instruction setting information. For example, Fig.10 As shown, the user selects the first vehicle as the first device, then Fig.11 As shown, the intelligent agent dialog box 1100 can output an inquiry message 1110 such as "Can you tell me what I need to notify you to do when the first vehicle does it?", and the user can input a reply message 1120 such as "When the first vehicle stops at the highway toll station, help me open the payment interface of the payment application." At this time, the first electronic device can determine the reply message 1120 as instruction setting information.

[0091] Through the above method, before outputting the inquiry information to the user, the user is first asked to select a specified device, which can ensure that the instruction setting information finally obtained is related to the device that has been connected to the first electronic device, so that the instructions generated based on the instruction setting information can be implemented, reducing the situation where the device corresponding to the instruction setting information is not a device connected to the first electronic device, and improving the accuracy of the instructions finally generated.

[0092] In other embodiments of the present application, when the first type is a connection type, the method of obtaining the first type of instruction setting information can also be similar to the method of obtaining the audio type of instruction setting information, that is, directly outputting the relevant inquiry information through the intelligent agent dialog box, and then determining the reply information of the user based on the inquiry information as the instruction setting information. For example, when the first type is a connection type, the first electronic device outputs inquiry information such as "Can you tell me what device needs to be notified to do what when doing what" to the user through the intelligent agent dialog box, and the user can input reply information such as "When the first vehicle stops at the highway toll station, help me open the payment application payment code" in the intelligent agent dialog box based on the inquiry information. At this time, the first electronic device can determine the reply information as the instruction setting information. This method does not require the user to select a specified device, which can reduce the user's operations.

[0093] Step 320: Generate a second instruction of the first type based on the instruction setting information.

[0094] In some embodiments of the present application, an agent in the first electronic device may generate a second instruction of the first type based on the acquired instruction setting information. The agent may use a machine learning algorithm such as a neural network, a rule engine, or a knowledge graph used for speech recognition, image recognition, etc. in deep learning to make inferences and decisions based on the instruction setting information, thereby obtaining instruction-related information, and then generating corresponding instructions based on the instruction-related information. For example, the agent obtains instruction-related information such as instruction execution conditions and instruction-associated actions from the instruction setting information through a neural network, and then generates corresponding instructions based on the instruction-related information.

[0095] In some embodiments of the present application, different types of instructions may be recorded in different formats. For example, the format of audio instructions may be as follows: Figure 5 As shown, it includes time conditions, positioning conditions, heard content, execution work and other contents. Time conditions, positioning conditions and heard content are combined into instruction execution conditions, and execution work is instruction-related actions. The format of display instructions can be as follows Figure 6 As shown in the figure, it includes seeing conditions, seeing content and executing work. Seeing conditions and seeing content together form the instruction execution conditions, and executing work is the instruction-related action. The format of the connection instruction can be as follows: Figure 7 As shown, it includes connection devices, connection content, execution work and other contents. The connection devices and connection content are combined into instruction execution conditions, and the execution work is the instruction-associated action.

[0096] Step 330: Store the second instruction into the instruction set.

[0097] In some embodiments of the present application, considering that the second instruction generated based on the instruction setting information may not be accurate enough, there may be errors with the instruction actually wanted by the user, and it does not meet the user's expectations. Therefore, in order to improve the accuracy of the instruction finally stored in the instruction set, before storing the second instruction in the instruction set, the generated instruction can be displayed to the user to let the user confirm whether the generated second instruction meets the user's expectations. When the user determines that the instruction meets the expectations, the user performs a confirmation operation on the second instruction. When the first electronic device receives the user's confirmation operation on the second instruction, it determines that the second instruction meets the user's expectations, thereby storing the generated second instruction in the instruction set and making the second instruction effective. Accordingly, if the generated second instruction does not meet the user's expectations, the user can delete the second instruction, thereby avoiding the second instruction that does not meet the user's expectations from being stored in the instruction set.

[0098] In some embodiments of the present application, the above-mentioned confirmation operation is an operation used to indicate that the second instruction meets the user's expectations. Exemplarily, the confirmation operation includes but is not limited to the user's click input of the confirmation control corresponding to the second instruction through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements and are not limited in the embodiments of the present invention. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0099] For example, see Fig.12 After generating the second instruction of the first type, the first electronic device displays the generated second instruction to the user through the agent dialog box 1200, so that the user can confirm whether the second instruction is effective based on the displayed content, such as Fig.12 As shown, the generated second instruction is displayed in the dialog box 1200 through the display control 1210. In addition, the display control 1210 also includes a cancel control 1211 and a confirmation control 1212. If the user confirms that the generated second instruction is accurate, the confirmation control 1212 is clicked, and the first electronic device determines that the second instruction displayed in the display control 1200 is effective in response to the user's click operation on the confirmation control 1212, and stores the second instruction in the instruction set. If the user confirms that the generated second instruction is inaccurate, the cancel control 1211 is clicked, and the first electronic device determines that the second instruction corresponding to the instruction-related content displayed in the display control 1200 is invalid in response to the user's click operation on the cancel control 1211, and deletes the second instruction. In this way, the instructions stored in the instruction set can be instructions that meet the user's expectations, and the instructions that do not meet the user's expectations can be eliminated.

[0100] Through the above technical solution, users can customize various types of commands such as hearing, seeing, connecting, etc. according to the command category, and can make personalized settings according to personal needs and preferences to meet the unique needs of different users.

[0101] In other embodiments, see Fig.13 Before the above step 110, the first electronic device can edit the instruction set through the following steps 1310-1330, which are described in detail below.

[0102] Step 1310: In response to a second operation on the first position, obtain second data.

[0103] In some embodiments of the present application, the second data includes at least one of the following: audio data, display content data, and connection data, wherein the connection data includes relevant information of a device connected to the first electronic device.

[0104] In some embodiments of the present application, the second operation is an operation of turning on the agent of the first electronic device. Exemplarily, the second operation includes but is not limited to a click input of the user to the first position through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and the embodiments of the present invention are not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a dragging gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, the above-mentioned second operation can be: a long press operation of the user to the first position through a finger. In response to the second operation, the first electronic device turns on the agent, and obtains the second data through the agent control data acquisition module.

[0105] Step 1320. Generate at least one instruction based on the second data.

[0106] In some embodiments of the present application, the first electronic device can generate at least one instruction based on the second data through an agent. Among them, the agent can use machine learning algorithms such as neural networks, rule engines, or knowledge graphs used for speech recognition, image recognition, etc. in deep learning to reason and make decisions based on instruction setting information, thereby obtaining information related to the instruction, and then generating corresponding instructions based on the instruction related information. For example, the agent obtains instruction related information such as instruction content, instruction execution conditions, and instruction-related actions from the instruction setting information through a neural network, and then generates corresponding instructions based on the instruction related information.

[0107] Step 1330. In response to a confirmation operation on a third instruction in the at least one instruction, store the third instruction into an instruction set, where the third instruction is any instruction in the at least one instruction.

[0108] In some embodiments of the present application, considering that the instructions generated based on the instruction setting information may not be accurate enough, there may be errors with the instructions actually wanted by the user, and they may not meet the user's expectations. Therefore, in order to improve the accuracy of the instructions finally stored in the instruction set, before storing at least one generated instruction in the instruction set, at least one generated instruction is first displayed to the user to allow the user to confirm whether the generated instruction meets the user's expectations. The user determines a third instruction that meets the expectations from the at least one instruction and performs a confirmation operation on the third instruction. Upon receiving the user's confirmation operation on the instruction, the first electronic device determines that the third instruction meets the user's expectations, thereby storing the third instruction in the instruction set to make the third instruction effective.

[0109] In some embodiments of the present application, the above-mentioned confirmation operation is used to indicate that the third instruction meets the user's expectations. Exemplarily, the confirmation operation includes but is not limited to the user's click input of the confirmation control corresponding to the third instruction through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements and are not limited in the embodiments of the present invention. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0110] In some embodiments of the present application, the first electronic device may generate multiple instructions based on the second data. When the first electronic device generates multiple instructions based on the second data, when displaying the generated instructions to the user for the user to confirm whether the instructions meet expectations, the first electronic device may generate multiple instructions based on the second data. Fig.14 The display control 1410 shown folds and displays multiple instructions, which can reduce screen occlusion. The display control 1410 can display the number of instructions so that the user can intuitively understand the number of instructions that need to be confirmed. Fig.14 Take the number of instructions as 4 as an example. When the user confirms the instruction, the display control 1410 may be clicked, and the first electronic device may display the following in response to the user's click operation on the display control 1410: Fig.15 The instruction list 1500 shown in FIG. 1 includes instruction identifiers of multiple instructions generated. The instruction identifiers may include text, patterns, etc. for indicating the content of the instructions, for example: Fig.15As shown, it contains instruction identifiers of 4 instructions, and the instruction identifier of each instruction includes a pattern and text corresponding to the instruction, wherein the pattern can be used to indicate the device corresponding to the instruction, and the text can include a summary of the instruction. The user can simply understand the content of the instruction based on the instruction identifier. In order to more accurately determine whether the instruction meets expectations, the user can also view the details of the instruction by clicking on the instruction identifier in the instruction list 1500. For example, when the user wants to view the first instruction in the instruction list 1500, the user can click on the first instruction in the instruction list 1500. The first electronic device responds to the user's click operation on the instruction by Fig.16 The display control 1610 shown shows the details of the instruction. Fig.15 As shown, the instruction list 1500 also includes a confirmation control 1510 corresponding to each instruction. For each instruction in the instruction list 1500, when the user determines that the instruction meets expectations, the user can perform a confirmation operation on the confirmation control 1510 corresponding to the instruction, so that the instruction takes effect and the instruction is stored in the instruction set. The confirmation operation can be a check operation on the confirmation control 1510. For example, Fig.15 As shown, if the user checks the confirmation control 1510 corresponding to the first instruction, the first electronic device determines that the first instruction meets the user's expectations, stores the instruction in the instruction set, and makes the instruction effective. For instructions that the user has not checked, it can be confirmed as an invalid instruction and deleted. In this way, the instructions stored in the instruction set can be instructions that meet the user's expectations, and instructions that do not meet the user's expectations can be eliminated.

[0111] In the above manner, the first electronic device can automatically obtain the instruction setting information and generate instructions without the user having to input the instruction setting information, thereby reducing the user's operations.

[0112] In some embodiments, the action associated with the first instruction may also be executed by a device connected to the first electronic device. Fig.17 As shown, after the above step 110, the first electronic device may further perform steps 130 to 140, which are described in detail below.

[0113] Step 130: In response to a third operation on the first instruction, display a device list, the device list including devices connected to the first electronic device.

[0114] In some embodiments of the present application, the third operation is an operation for indicating forwarding the first instruction. Exemplarily, the third operation includes but is not limited to the user's pull-down input or click input of the first instruction through a touch device such as a finger or a stylus, or a voice instruction input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and the embodiments of the present invention are not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, when the user needs to forward the first instruction to a device connected to the first electronic device, the user can pull down the first instruction displayed in the first position.

[0115] In response to the user's third operation on the first instruction, the first electronic device determines that the user needs to forward the first instruction, and displays a device list so that the user can select a device to which the first instruction is to be forwarded based on the device list.

[0116] In some embodiments of the present application, in order to enable the forwarded device to execute the first instruction, the device list may only include the device identifiers of the devices connected to the first electronic device and capable of executing the first action. For example, the first action is to open the map application to navigate to Gaolaozhuang. The tablet computer connected to the first electronic device, the first vehicle, and the second vehicle are all installed with the map application, and both can navigate through the map application. Fig.18 As shown, the device list includes the device identification of the tablet computer, the device identification of the first vehicle, and the device identification of the second vehicle.

[0117] Step 140: In response to a selection operation on a second device in the device list, a first instruction is sent to the second device, where the second device is any device in the device list.

[0118] In some embodiments of the present application, the above-mentioned selection operation is used to select the second device to which the user wants to forward the first instruction. Exemplarily, the above-mentioned selection operation includes but is not limited to the user's click input of the target device identifier in the device list through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which can be determined according to actual usage requirements and is not limited by the embodiments of the present invention. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, such as Fig.18 As shown, if the user wants to forward the first instruction to the second vehicle, the user can click the device identifier of the second vehicle in the device list by a finger.

[0119] In response to the user's selection operation on the second device, the first electronic device determines that the second device is the device to which the first instruction is to be forwarded, and sends the first instruction to the second device so that the second device can perform a first action associated with the first instruction.

[0120] In some embodiments of the present application, in order to improve the security of interaction and reduce the adverse effects of malicious instructions on devices, after receiving the first instruction, the second device may first ask the user whether to execute the first action associated with the first instruction, and when the user confirms that the first action can be executed, the second device executes the first action. In this way, the interference of malicious instructions can be reduced.

[0121] Through the above technical solution, command interaction between the first electronic device and the device connected to it can be achieved without complicated settings and operations, which improves the connection speed and compatibility between the devices and achieves seamless connection and efficient interaction.

[0122] The interactive method provided in the embodiment of the present application can be executed by an interactive device. In the embodiment of the present application, the interactive method executed by the interactive device is taken as an example to illustrate the interactive device provided in the embodiment of the present application.

[0123] See also Fig.19 , is a schematic diagram of the structure of an interactive device provided in an embodiment of the present application, wherein the interactive device 1900 is applied to a first electronic device, such as Fig.19 As shown, the interactive device 1900 includes the following modules:

[0124] Display module 1910, configured to display the first instruction at a first position on the screen when the acquired first data matches a first instruction in the instruction set, wherein the instruction set includes at least one instruction, and the at least one instruction includes the first instruction;

[0125] The processing module 1920 is configured to execute a first action associated with the first instruction in response to a first operation on the first instruction.

[0126] The interactive device provided in the embodiment of the present application autonomously acquires first data, matches the first data with a pre-edited instruction set, displays a first instruction matching the first data at a first position on the screen, and executes a first action associated with the first instruction in response to a first operation on the first instruction. According to the embodiment of the present application, the electronic device autonomously acquires data during operation, and determines whether there is a matching instruction based on the acquired data. If there is a matching instruction, the instruction is displayed at the first position of the screen, so that the user only needs to perform a first operation on the displayed instruction to make the electronic device perform the corresponding action, without having to search for the corresponding function in a complex menu, making the interaction simpler and more convenient.

[0127] In some embodiments, the first data includes at least one of the following: audio data, display content data, and connection data, and the connection data includes relevant information of a device connected to the first electronic device.

[0128] In some embodiments, the apparatus 1900 further includes:

[0129] an acquisition module, configured to acquire instruction setting information of the first type in response to a selection operation of the first type in the instruction type option before displaying the first instruction at a first position on the screen when the acquired first data matches the first instruction in the instruction set;

[0130] The processing module 1920 is further configured to:

[0131] Based on the instruction setting information, generate a second instruction of the first type;

[0132] The second instruction is stored in the instruction set.

[0133] In some embodiments, the first type is a type corresponding to audio data, and the acquisition module is specifically used to:

[0134] Outputting inquiry information for inquiring instruction setting information through the agent dialog box;

[0135] Obtain the answer information entered by the user in the agent dialog box based on the query information;

[0136] The reply information is determined as the first type of instruction setting information.

[0137] In some embodiments, the first type is a type corresponding to display content data, and the acquisition module is specifically used to:

[0138] In response to a user input to a first area of ​​the screen, acquiring display content data of the first area;

[0139] The display content data of the first area is determined as the first type of instruction setting information.

[0140] In some embodiments, the first type is a type corresponding to the connection data, and the display module is further configured to:

[0141] displaying a device list, the device list including devices connected to the first electronic device;

[0142] Get the module, specifically for:

[0143] In response to a selection operation on a first device in the device list, obtaining dialogue information corresponding to the first device in the agent dialog box, where the first device is any device in the device list;

[0144] The dialog information is determined as the first type of instruction setting information.

[0145] In some embodiments, the apparatus 1900 further includes:

[0146] an acquisition module, configured to acquire second data before displaying the first instruction at a first position on the screen when the acquired first data matches the first instruction in the instruction set, the second data comprising at least one of the following: audio data, display content data, and connection data, wherein the connection data comprises relevant information of a device connected to the first electronic device;

[0147] The processing module 1920 is further configured to:

[0148] generating at least one instruction based on the second data;

[0149] In response to a confirmation operation on a third instruction in the at least one instruction, the third instruction is stored in the instruction set, where the third instruction is any instruction in the at least one instruction.

[0150] In some embodiments, the display module 1910 is further configured to:

[0151] In a case where the acquired first data matches a first instruction in the instruction set, after displaying the first instruction at a first position on the screen, in response to a third operation on the first instruction, displaying a device list, the device list including devices connected to the first electronic device;

[0152] The apparatus 1900 further includes:

[0153] The sending module is used to send a first instruction to the second device in response to a selection operation on the second device in the device list, where the second device is any device in the device list.

[0154] The interactive device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0155] The interactive device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0156] The interactive device provided in the embodiment of the present application can achieve Figures 1 to 18 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0157] Alternatively, if Fig. 20 As shown, an embodiment of the present application also provides an electronic device 2000, including a processor 2001 and a memory 2002, wherein the memory 2002 stores programs or instructions that can be executed on the processor 2001, and when the program or instructions are executed by the processor 2001, the various steps of the above-mentioned interaction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0158] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0159] Fig.21 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0160] The electronic device 2100 includes but is not limited to: a radio frequency unit 2101, a network module 2102, an audio output unit 2103, an input unit 2104, a sensor 2105, a display unit 2106, a user input unit 2107, an interface unit 2108, a memory 2109, and a processor 2110 and other components.

[0161] Those skilled in the art will appreciate that the electronic device 2100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 2110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.21 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0162] The display unit 2106 is configured to display the first instruction at a first position on the screen when the acquired first data matches a first instruction in the instruction set, wherein the instruction set includes at least one instruction, and the at least one instruction includes the first instruction;

[0163] The processor 2110 is configured to execute a first action associated with the first instruction in response to a first operation on the first instruction.

[0164] The electronic device provided in the embodiment of the present application autonomously acquires first data, matches the first data with a pre-edited instruction set, displays a first instruction matching the first data at a first position on the screen, and executes a first action associated with the first instruction in response to a first operation on the first instruction. According to the embodiment of the present application, the electronic device autonomously acquires data during operation, and determines whether there is a matching instruction based on the acquired data, and displays the instruction at the first position of the screen when there is a matching instruction, so that the user only needs to perform a first operation on the displayed instruction to make the electronic device execute the corresponding action, without having to search for the corresponding function in a complex menu, making the interaction simpler and more convenient.

[0165] In some embodiments, the first data includes at least one of the following: audio data, display content data, and connection data, and the connection data includes relevant information of a device connected to the first electronic device.

[0166] In some embodiments, the input unit 2104 is used to:

[0167] In a case where the acquired first data matches a first instruction in the instruction set, before displaying the first instruction at a first position on the screen, in response to a selection operation on a first type in the instruction type option, acquiring instruction setting information of the first type, wherein the first type is any type in the instruction type option;

[0168] The processor 2110 is further configured to:

[0169] Based on the instruction setting information, generate a second instruction of the first type;

[0170] Storing the second instruction in the instruction set. ;

[0171] In some embodiments, the first type is a type corresponding to audio data, and the input unit 2104 is specifically used to:

[0172] Outputting inquiry information for inquiring instruction setting information through the agent dialog box;

[0173] Obtain the answer information entered by the user in the agent dialog box based on the query information;

[0174] The reply information is determined as the first type of instruction setting information.

[0175] In some embodiments, the first type is a type corresponding to the displayed content, and the input unit 2104 is specifically used to:

[0176] In response to a user input to a first area of ​​the screen, acquiring display content data of the first area;

[0177] The display content data of the first area is determined as the first type of instruction setting information.

[0178] In some embodiments, the first type is a type corresponding to the connection data, and the display unit 2106 is further configured to:

[0179] displaying a device list, the device list including devices connected to the first electronic device;

[0180] The input unit 2104 is specifically used for:

[0181] In response to a selection operation on a first device in the device list, obtaining dialogue information corresponding to the first device in the agent dialog box, where the first device is any device in the device list;

[0182] The dialog information is determined as the first type of instruction setting information.

[0183] In some embodiments, the input unit 2104 is further used to:

[0184] an acquisition module, configured to acquire second data before displaying the first instruction at a first position on the screen when the acquired first data matches the first instruction in the instruction set, the second data comprising at least one of the following: audio data, display content data, and connection data, wherein the connection data comprises relevant information of a device connected to the first electronic device;

[0185] The processor 2110 is further configured to:

[0186] generating at least one instruction based on the second data;

[0187] In response to a confirmation operation on a third instruction in the at least one instruction, the third instruction is stored in the instruction set, where the third instruction is any instruction in the at least one instruction.

[0188] In some embodiments, the display unit 2106 is further configured to:

[0189] In a case where the acquired first data matches a first instruction in the instruction set, after displaying the first instruction at a first position on the screen, in response to a third operation on the first instruction, displaying a device list, the device list including devices connected to the first electronic device;

[0190] The interface unit 2108 is used to send a first instruction to the second device in response to a selection operation on the second device in the device list, where the second device is any device in the device list.

[0191] It should be understood that in the embodiment of the present application, the input unit 2104 may include a graphics processor (Graphics Processing Unit, GPU) 21041 and a microphone 21042, and the graphics processor 21041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 2106 may include a display panel 21061, and the display panel 21061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 2107 includes a touch panel 21071 and at least one of other input devices 21072. The touch panel 21071 is also called a touch screen. The touch panel 21071 may include two parts: a touch detection device and a touch controller. Other input devices 21072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0192] The memory 2109 can be used to store software programs and various data. The memory 2109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 2109 may include a volatile memory or a non-volatile memory, or the memory 2109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 2109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0193] The processor 2110 may include one or more processing units; optionally, the processor 2110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 2110.

[0194] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned interaction method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0195] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0196] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned interaction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0197] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0198] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned interaction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0199] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0200] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0201] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An interactive method, characterized in that: Applied to a first electronic device, the method includes: In a case where the acquired first data matches a first instruction in an instruction set, displaying the first instruction at a first position on the screen, wherein the instruction set includes at least one instruction, and the at least one instruction includes the first instruction; In response to a first operation on the first instruction, a first action associated with the first instruction is performed.

2. The method according to claim 1, characterized in that The first data includes at least one of the following: audio data, display content data, and connection data, and the connection data includes relevant information of a device connected to the first electronic device.

3. The method according to claim 1, characterized in that In the case where the acquired first data matches the first instruction in the instruction set, before displaying the first instruction at the first position of the screen, the method further includes: In response to a selection operation of a first type in the instruction type option, obtaining instruction setting information of the first type, where the first type is any type in the instruction type option; Based on the instruction setting information, generate a second instruction of the first type; The second instruction is stored in the instruction set.

4. The method according to claim 3, characterized in that The first type is a type corresponding to audio data, and the obtaining instruction setting information of the first type includes: Outputting inquiry information for inquiring instruction setting information through the agent dialog box; Obtaining the answer information input by the user in the agent dialog box based on the inquiry information; The reply information is determined as the first type of instruction setting information.

5. The method according to claim 3, characterized in that: The first type is a type corresponding to the display content data, and the obtaining of the instruction setting information of the first type includes: In response to a user input to a first area of ​​the screen, acquiring display content data of the first area; The display content data of the first area is determined as the first type of instruction setting information.

6. The method according to claim 3, characterized in that The first type is a type corresponding to the connection data, and the obtaining of the instruction setting information of the first type includes: displaying a device list, wherein the device list includes devices connected to the first electronic device; In response to a selection operation of a first device in the device list, obtaining dialogue information corresponding to the first device in the agent dialog box, wherein the first device is any device in the device list; The dialog information is determined as the first type of instruction setting information.

7. The method according to claim 1, characterized in that In the case where the acquired first data matches the first instruction in the instruction set, before displaying the first instruction at the first position of the screen, the method further includes: In response to a second operation on the first position, obtaining second data, the second data comprising at least one of the following: audio data, display content data, and connection data, wherein the connection data comprises relevant information of a device connected to the first electronic device; generating at least one instruction based on the second data; In response to a confirmation operation on a third instruction in the at least one instruction, the third instruction is stored in the instruction set, and the third instruction is any instruction in the at least one instruction.

8. The method according to any one of claims 1 to 7, characterized in that: In the case where the acquired first data matches the first instruction in the instruction set, after displaying the first instruction at the first position of the screen, the method further includes: In response to a third operation on the first instruction, displaying a device list, the device list including devices connected to the first electronic device; In response to a selection operation of a second device in the device list, the first instruction is sent to the second device, where the second device is any device in the device list.

9. An interactive device, characterized in that: Applied to a first electronic device, the device comprises: a display module, configured to display a first instruction in an instruction set at a first position on a screen when the acquired first data matches the first instruction in the instruction set, wherein the instruction set includes at least one instruction, and the at least one instruction includes the first instruction; The processing module is configured to execute a first action associated with the first instruction in response to a first operation on the first instruction.

10. The device according to claim 9, characterized in that The device also includes: an acquisition module, configured to acquire second data in response to a second operation on a first position of a screen before displaying the first instruction at the first position of the screen if the acquired first data matches the first instruction in the instruction set, wherein the second data includes at least one of the following: audio data, display content data, and connection data, wherein the connection data includes relevant information of a device connected to the first electronic device; The processing module is further used for: generating at least one instruction based on the second data; In response to a confirmation operation on a third instruction in the at least one instruction, the third instruction is stored in the instruction set, and the third instruction is any instruction in the at least one instruction.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the interaction method according to any one of claims 1 to 8 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the interactive method according to any one of claims 1 to 8 are implemented.

Images