Equipment control method, electronic equipment, readable storage medium and chip

Abstract

The invention provides a device control method, an electronic device, a readable storage medium and a chip, and relates to the technical field of artificial intelligence. The method comprises the following steps: displaying a first interface, wherein the first interface comprises a target area in which information is to be filled; in response to a first user instruction, automatically filling first target information in the target area; wherein the first target information is determined from memory information stored in the terminal equipment according to the first user instruction and interface information of the first interface. In the technical scheme provided by the invention, the terminal equipment can automatically fill information in the display interface according to the user instruction, the interface information of the display interface and the user memory information stored in the terminal equipment, so that the terminal equipment has better user experience.

Description

[0001] This application is a divisional application. The application number of the parent application is 202411091268.0, and the application date of the parent application is August 8, 2024. The parent application claims the priority of the prior application 202410809043.8, and the priority date is June 20, 2024. The entire contents of the parent application are incorporated by reference in this application.

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 20, 2024, with application number 202410809043.8 and application name “Device control method, electronic device, readable storage medium and chip”, the entire contents of which are incorporated by reference in this application. Technical Field

[0003] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a device control method, an electronic device, a readable storage medium and a chip. Background Art

[0004] With the development of computer technology, voice assistants such as "Xiaoyi", "YOYO", and "Siri" have gradually appeared on terminal devices such as mobile phones and tablets. Users can interact with terminal devices through voice assistants. For example, users can use voice assistants to control terminal devices to make calls, send text messages, set alarms, play music, plan navigation, or control smart home devices. However, terminal devices currently only support processing simpler user instructions such as making calls through voice assistants. If the user instructions given by the user are more complex, such as "please fill out this form for me", "please order me a milk tea takeaway", etc., the terminal device will not be able to execute the user instruction, resulting in a poor user experience. Summary of the invention

[0005] The present application provides a device control method, an electronic device, a readable storage medium and a chip, which are used to solve the problem in the prior art that terminal devices cannot execute complex user instructions.

[0006] In a first aspect, an embodiment of the present application provides a device control method, which is applied to a terminal device and specifically includes: displaying a first interface, the first interface including a target area for information to be filled in; in response to a first user instruction, automatically filling in first target information in the target area; wherein the first target information is determined from memory information stored in the terminal device based on the first user instruction and interface information of the first interface.

[0007] Through the device control method provided in this embodiment, the terminal device can perceive and understand the interface information of the display interface, and according to the interface information and user instructions, obtain the corresponding target information from the user memory information stored in the terminal device, and automatically fill in the target information in the display interface, providing a better user experience.

[0008] In some embodiments, the first user instruction includes: a voice control instruction; or a text control instruction; or a selection operation on a first control, the first control is displayed based on an index word entered in a target area, and the index word is associated with the first target information.

[0009] Exemplarily, the index words include: person names, event names, place names, keywords, product names, dates, etc. Taking the index word being a person name as an example, the first target information is the user information corresponding to the person name.

[0010] In some embodiments, the first user instruction includes a selection operation on the first control, and in response to the first user instruction, the first target information is automatically filled in the target area, including: displaying index words in the input method interface according to the input operation in the input method application; determining the index words as the first control; and automatically filling in the first target information in the target area in response to the selection operation on the first control.

[0011] Through the method provided in this embodiment, the user can quickly control the terminal device to automatically input the target information in the display interface through the input method application. Taking the index word as the name "Zhang San" as an example, the user can enter the name "Zhang San" in the input method application to automatically fill in the user information corresponding to "Zhang San" into the target area of ​​the display interface, which provides a good user experience.

[0012] In some embodiments, the first user instruction includes a selection operation on the first control, and in response to the first user instruction, the first target information is automatically filled in the target area, including: filling in the index word in the target area of ​​the first interface according to the input operation; displaying the first control on the first interface according to the index word; and automatically filling in the first target information in the target area in response to the selection operation on the first control.

[0013] In this embodiment, the first control displayed according to the index word can be located at any position of the target area, for example, near the filled index word, and this embodiment does not limit this.

[0014] In some embodiments, in response to a first user instruction, the first target information is automatically filled in the target area, including: in response to the first user instruction, candidate information is displayed on the first interface, and the candidate information is determined from the memory information stored in the terminal device according to the first user instruction and the interface information of the first interface; according to the control operation, the first target information is determined from the candidate information; and the first target information is automatically filled in the target area.

[0015] Through the method provided in the embodiment of the present application, the user can select and modify the candidate information before the terminal device automatically fills in the target information, that is, the information is automatically filled in after the user confirms the information. It can be understood that this method can ensure the accuracy of information filling.

[0016] In some embodiments, the method further includes: receiving a pause instruction during the process of automatically filling in the first target information in the target area; in response to the pause instruction, the terminal device pauses filling in the first target information. That is, during the process of the terminal device automatically filling in the information, the user can control the terminal device to pause filling at any time according to needs, which has a better user experience.

[0017] In some embodiments, the method further includes: after pausing to fill in the first target information, or after completing the first target information, receiving a second user instruction; in response to the second user instruction, automatically replacing the filled first target information with the second target information; wherein the second target information is determined from the memory information stored in the terminal device according to the second user instruction and the interface information of the first interface. That is, the terminal device can automatically fill in the information again according to the user instruction.

[0018] In some embodiments, before automatically filling in the first target information in the target area in response to the first user instruction, the method further includes: receiving an information storage instruction, the information storage instruction carrying an indication of the first target information; and storing the first target information. Exemplarily, the information storage instruction may be a voice control instruction or a text control instruction, etc., and the terminal device may store the information carried in the information storage instruction as user memory information.

[0019] In some embodiments, before automatically filling in the first target information in the target area in response to the first user instruction, the method further includes: displaying a second interface, the second interface including the first target information; receiving an information storage instruction, the information storage instruction is used to instruct the terminal device to store the first target information; storing the first target information in the memory information. That is, the terminal device can store the information displayed in its interface as user memory information according to the user's control operation.

[0020] In a second aspect, an embodiment of the present application provides a device control method, which is applied to a terminal device, and the method includes: obtaining a first user instruction, the first user instruction is used to control the terminal device to perform a target operation; in response to the first user instruction, automatically executing the target operation according to the interface information and first target information of the terminal device, wherein the first target information is obtained from memory information stored in the terminal device according to the first user instruction.

[0021] Through the device control method provided in this embodiment, after receiving user instructions, the terminal device can determine and execute target operations that meet the actual needs of the user based on the interface information of the display interface and the user memory information stored in the terminal device, thereby accurately executing user instructions and providing a better user experience.

[0022] In some embodiments, the target operation includes purchasing a target product, and the first target information includes product information of the target product, such as product name, quantity, product attributes, application identifier, and delivery address; in response to a first user instruction, the target operation is automatically executed according to the interface information of the terminal device and the first target information, including: in response to the first user instruction, a purchase order for the target product is generated according to the interface information of the terminal device and the product information of the target product.

[0023] In other words, the terminal device can help users automatically purchase goods based on user instructions and interface information, combined with user memory information stored in the terminal device, such as helping users order takeout, providing a better user experience.

[0024] In some embodiments, the target operation includes filling in information, and the first target information includes content to be filled in.

[0025] In some embodiments, before obtaining the first user instruction, the method also includes: displaying a first interface, the first interface including a target area for information to be filled in; and, in response to the first user instruction, automatically performing a target operation according to the interface information of the terminal device and the first target information, including: in response to the first user instruction, automatically filling in the first target information in the target area according to the interface information of the terminal device, wherein the first target information is determined from the memory information stored in the terminal device according to the first user instruction and the interface information of the first interface.

[0026] In this embodiment, the terminal device can perceive and understand the interface information of the display interface, and obtain target information matching the interface information from the user memory information stored in the terminal device according to the user instructions and interface information, and automatically fill in the target information in the display interface.

[0027] In some embodiments, in response to a first user instruction, a target operation is automatically performed according to the interface information of the terminal device and the first target information, including: in response to the first user instruction, a first interface is displayed, the first interface including a target area for information to be filled in; according to the interface information of the first interface, the first target information is automatically filled in the target area, wherein the first target information is determined from the memory information stored in the terminal device according to the first user instruction and the interface information of the first interface.

[0028] In some embodiments, automatically filling in the first target information in the target area includes: displaying candidate information in the first interface, wherein the candidate information is determined from memory information stored in the terminal device based on a first user instruction and interface information of the first interface; determining the first target information from the candidate information based on a control operation; and automatically filling in the first target information in the target area.

[0029] In some embodiments, the method also includes: after pausing to fill in the first target information, or after completing the first target information, receiving a second user instruction; in response to the second user instruction, automatically replacing the filled in first target information with the second target information; wherein the second target information is determined from the memory information stored in the terminal device based on the second user instruction and the interface information of the first interface.

[0030] In some embodiments, the method further includes: receiving a pause instruction during the process of automatically executing the target operation according to the first target information and the interface information of the terminal device; and pausing the execution of the target operation in response to the pause instruction.

[0031] In some embodiments, before receiving the first user instruction, the method further includes: receiving an information storage instruction, the information storage instruction carrying the first target information; and storing the first target information.

[0032] In some embodiments, before receiving the first user instruction, the method also includes: displaying a second interface, the second interface including the first target information; receiving an information storage instruction, the information storage instruction is used to instruct to store the first target information; and storing the first target information.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors; the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the electronic device to execute the method shown in the first or second aspect above.

[0034] In a fourth aspect, an embodiment of the present application provides a chip, which is applied to an electronic device. The chip includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method shown in the first or second aspect above.

[0035] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method shown in the first aspect or the second aspect is implemented.

[0036] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device implements the method shown in the first aspect or the second aspect above.

[0037] It can be understood that the beneficial effects of the third to sixth aspects mentioned above can be found in the relevant descriptions in the first or second aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of a voice assistant startup scenario provided by an embodiment of the present application;

[0039] Figure 2 is a schematic diagram of a voice control scenario provided by an embodiment of the present application;

[0040] Figure 3 It is a schematic diagram of a device control system to which the device control method provided in this application is applicable;

[0041] Figure 4 is a flow chart of a device control method provided by an embodiment of the present application;

[0042] Figure 5 It is a schematic diagram of the process of generating structured user memory information provided by an embodiment of the present application;

[0043] Figure 6A to Figure 6D It is a schematic diagram of the storage process of user memory information provided by different embodiments of the present application;

[0044] Figure 7 is a schematic flow chart of a device control method provided by an embodiment of the present application;

[0045] Figure 8 to Figure 11 It is a schematic diagram of the process of automatically filling in information provided by different embodiments of the present application;

[0046] Fig.12 is a schematic diagram of an interface including candidate information provided in an embodiment of the present application;

[0047] Fig.13 It is a schematic diagram of the process of re-filling information provided in an embodiment of the present application;

[0048] Figure 14-15 is a schematic diagram of the process of automatically purchasing goods provided by different embodiments of the present application;

[0049] Fig.16 is a structural schematic diagram of an electronic device provided by this embodiment;

[0050] Fig.17 It is a schematic diagram of the structure of a chip provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solution provided in the embodiments of the present application is described below with reference to the accompanying drawings.

[0052] It should be understood that in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0053] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0054] With the development of computer technology, voice assistants such as "Xiaoyi", "YOYO" and "Siri" have gradually appeared on mobile phones, tablet computers and other terminal devices to assist users in human-computer interaction.

[0055] In this embodiment, the voice assistant may also be referred to as an AI assistant, intelligent voice, digital assistant, virtual assistant, etc. It can be understood as an information processing system that can recognize natural language input in the form of voice and / or text to infer user intent, and perform corresponding actions based on the inferred user intent. In addition, the system can respond to user input through audible (such as voice) and visual (such as text, pictures) output forms.

[0056] In order to save power consumption of terminal devices, voice assistants are usually in a dormant state. Users can start the voice assistant through specific control operations. Starting the voice assistant can also be called waking up the voice assistant. For example Figure 1As shown, the user can start the voice assistant by saying the wake-up words such as "Xiaoyi Xiaoyi" or "Hello YOYO". Alternatively, the voice assistant can be started by clicking the voice assistant icon. Alternatively, the voice assistant can be started by long pressing the power button. This embodiment does not limit the way to start the voice assistant.

[0057] After the terminal device starts the voice assistant, it can use the voice assistant to perform corresponding operations according to the user's instructions. For example, according to the voice command "play song A" input by the user, the terminal device is controlled to play song A (see Figure 2 As shown). Or, according to the voice command "Call Lucy" input by the user, the contact Lucy's phone number is dialed. Or, according to the voice input by the user "Wake me up at 7 o'clock tomorrow morning", the alarm clock for 7 o'clock tomorrow morning is set. Or, according to the voice input by the user "What holiday is today", a corresponding answer is provided, such as "Today is New Year's Day". It can be seen that the voice assistant significantly improves the convenience of user interaction with terminal devices and brings users a new human-computer interaction experience.

[0058] In this embodiment, user instructions include simple user instructions and complex user instructions. Among them, simple user instructions are also called first-class user instructions, which refer to user instructions that do not require understanding and perception of the display interface and can usually be completed using one control operation, such as instructions for turning on Bluetooth, making a call, sending text messages, playing music, or setting an alarm. Complex user instructions are also called second-class user instructions, which refer to user instructions that require understanding and perception of the display interface during the implementation process, or that can only be implemented through multiple control operations, such as "please fill out this form for me" and "please order me a takeaway milk tea".

[0059] At present, terminal devices only support processing simple user commands through voice assistants. If the user gives a complex user command, such as "help me fill out this form", since the user command does not indicate the relevant information required to execute the command (such as the information required when filling out the form), and the terminal device cannot understand and perceive the current interface information, the terminal device cannot dynamically generate a clear and executable processing plan for the user command, and thus cannot generate control operations that meet the user's actual needs. That is, the terminal device cannot correctly execute the user command, and the user experience is poor.

[0060] To this end, an embodiment of the present application provides a device control method, in which, for complex user instructions, the terminal device can combine interface information and user memory information to control the device, thereby achieving control operations that meet the actual needs of the user and improving the user experience.

[0061] To facilitate understanding, the relevant terms and concepts involved in the embodiments of the present application are first introduced below.

[0062] (1) Interface information

[0063] Interface information includes content information and layout information of the display interface. Content information includes application identification (ID), window ID, application activation function and other information related to the application. Layout information is used to represent the hierarchical structure of the display interface, control information, atomic operation properties of controls (such as whether the control is clickable, whether it is slidable, etc.), icons and text, etc.

[0064] (2) Memory information

[0065] In this embodiment, the memory information is historical information that the user authorizes the terminal device to obtain and store, and the memory information includes user memory information and interface fusion memory information.

[0066] User memory information refers to memory information related to user dialogue, user operation, user storage or user usage habits, such as card information, important date information, event information, etc. The terminal device can determine and store user memory information based on user instructions or by using active memory technology combined with a memory extraction model.

[0067] Interface fusion memory information refers to memory information related to the historical display interface, which includes historical interface information (such as content information, layout information), user instructions and control operation sequences related to the historical display interface, etc.

[0068] (3) User instructions

[0069] In this embodiment, the user instruction may also be referred to as a user request, a Query request (i.e., a query request), etc., including a voice control instruction, a text control instruction, a control operation, and the like.

[0070] The voice control instruction may be a wake-up instruction for waking up the voice assistant, or an operation instruction for controlling the terminal device to perform a target operation, such as "call Lucy for me" in voice form.

[0071] The text control command is usually used to control the terminal device to perform the target operation, and is input by the user through the user dialog box of the voice assistant. For example, the text control command can be in the form of text "please fill out this form for me", "please order a takeaway milk tea for me", "set an alarm for 7 o'clock tomorrow morning", etc.

[0072] The control operation may be a user pressing a physical key, such as a long press of a power key. In addition, the control operation may also be a touch operation on a control in the display interface, such as a click operation, a slide operation, etc.

[0073] (4) Real-time user instructions and non-real-time user instructions

[0074] In this embodiment, according to whether the user instruction requires the terminal device to immediately respond and output, this embodiment divides the user instruction into an immediate user instruction and a non-immediate user instruction.

[0075] Instant user commands, also known as instant queries, refer to commands that require immediate output responses from the terminal device, such as "Play a song for me", "Set the air conditioning temperature in the living room to 26°C", "Call Lucy", "Fill out this form for me", or "Order me a takeaway milk tea".

[0076] A non-immediate user command, also called a non-immediate query, refers to a command that the terminal device needs to process internally but does not need to output a response for the time being, such as "Please help me store Lucy's phone number 135****2195".

[0077] Based on the above basic conceptual information, the technical solution provided in the embodiments of the present application is described in detail below.

[0078] Figure 3 Schematic diagram of a device control system to which the device control method provided in the present application is applicable. The system includes a terminal device 100 and a server 200, and the terminal device 100 and the server 200 can be wirelessly connected via wireless fidelity (Wi-Fi), cellular communication connection, etc., or can be connected via a wired connection such as a network cable, which is not limited in this embodiment.

[0079] The terminal device 100, also referred to as the terminal side, includes but is not limited to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA) and other terminal devices. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0080] In this embodiment, the terminal device includes an instruction acquisition module 110 , an instruction parsing module 120 , a user memory module 130 , an interface fusion understanding module 140 , and an instruction execution module 150 .

[0081] Instruction acquisition module 110, used to obtain diversified user instructions, such as the voice control instructions, text control instructions and control operations shown in the above embodiments. After obtaining the user instructions, the instruction acquisition module 110 sends them to the instruction parsing module 120 for processing.

[0082] Instruction parsing module 120 , used to parse the meaning of the instruction and determine the type of instruction after receiving the user instruction. Among them, parsing the meaning of the instruction refers to determining the content of the user's instruction. Exemplarily, the instruction parsing module 120 can use its own capabilities to parse the user instruction, or it can use the capabilities of the server 200 to parse the user instruction, which is not limited in this embodiment. Determining the type of instruction refers to determining whether the user instruction belongs to the first category of user instructions or the second category of user instructions based on the user's instruction content.

[0083] In some embodiments, the instruction parsing module 120 may process different types of user instructions in different ways.

[0084] If the received user instruction is a first-class user instruction, the instruction parsing module 120 determines the corresponding control operation according to the user instruction, and sends the control operation to the instruction execution module 150 for processing. Exemplarily, the instruction parsing module 120 can determine the control operation corresponding to the user instruction by its own capabilities, or by the capabilities of the server 200. For example, if the user instruction is "turn on Bluetooth", the instruction parsing module 120 generates a control Bluetooth turn-on instruction, and sends the instruction to the instruction execution module 150 for processing.

[0085] If the received user instruction is a second type of user instruction, the instruction parsing module 120 may perform a corresponding operation according to the immediacy of the user instruction, as shown below.

[0086] When the user instruction is a non-immediate user instruction, such as "Help me remember my daughter's birthday, September 1st", the instruction parsing module 120 generates structured user memory information according to the user instruction. Exemplarily, the instruction parsing module 120 can generate structured user memory information through its own capabilities. Alternatively, the instruction parsing module 120 generates structured user memory information with the help of the capabilities of the server 200. For example, the instruction parsing module 120 sends the user instruction or the parsed user instruction to the server 200, and the server 200 converts it into structured user memory information through the structured memory module 210, and then sends it to the user memory module 130 of the terminal device 100 for storage (see below for details about the introduction of the structured memory module 210).

[0087] When the user instruction is an immediate user instruction but does not include memory content, for example, when the user instruction is "help me fill out this form", the instruction parsing module 120 notifies the interface fusion understanding module 140 to work. The specific working process of the interface fusion understanding module 140 is described below and will not be repeated here.

[0088] When the user instruction is an immediate user instruction and includes memory content, for example, when the user instruction is "Please fill in Lucy's phone number 135****2195 for me", the instruction parsing module 120 generates structured user memory information through its own / server 200 capabilities on the one hand, and notifies the interface fusion understanding module 140 to work on the other hand.

[0089] User memory module 130 , used to store structured user memory information. For details of the structured user memory information, please refer to the relevant description in the "structured memory module 210" of the "server 200" below, which is not limited here.

[0090] Interface Fusion Understanding Module 140 , including a content understanding unit 141, a layout understanding unit 142 and an interface memory unit 143. Among them, the interface memory unit 143 is an optional unit.

[0091] Content Understanding Unit 141 , used to generate content information of the display interface according to the control information and application programs in the display interface. It should be noted that, in the process of understanding the layout, if there are multiple windows in the display interface, the content understanding unit 141 can understand the content of each window separately.

[0092] Layout Understanding Unit 142 , used to identify objects in the current display interface, understand the relationship between each object, and parse the overall meaning of the display interface, and calmly map it into structured layout information. It should be noted that the layout understanding unit 142 can fully use its own capabilities to understand the layout, or it can use the capabilities of the server 200 to understand the layout in whole or in part, and this embodiment does not limit this. In the process of understanding the layout with the capabilities of the server 200, the layout understanding unit 142 sends the display interface to the server 200 and receives the layout information returned by the server.

[0093] In some embodiments, after respectively obtaining the content information and layout information of the display interface, the content understanding unit 141 and the layout understanding unit 142 send them together with the user instructions to the server 200, so that the server 200 determines and issues the corresponding control instruction sequence.

[0094] In some other embodiments, the interface fusion understanding module 140 further includes Interface memory unit 143The interface memory unit 143 stores interface fusion memory information of historical display interfaces.

[0095] Based on this, after generating the content information of the current display interface, the content understanding unit 141 can send a search instruction to the interface memory unit 143, and the search instruction carries the user instruction and the content information, and is used to instruct the interface memory unit 143 to search for the interface fusion memory information matching the current display interface. In addition, the layout understanding unit 142 also sends the layout information of the current display interface to the interface memory unit 143.

[0096] Accordingly, after receiving the search instruction and the layout information, the interface memory unit 143 fuses the user instruction in the search instruction, the content information of the current display interface, and the layout information of the display interface to obtain the interface fusion understanding information of the current display interface. Secondly, according to the current interface fusion understanding information, similar interface fusion memory information is searched in the interface memory unit 143. Then, the current interface fusion understanding information and the interface fusion memory information are sent to the server 200 together, so that the server 200 can determine and issue the corresponding control instruction sequence in combination with these two kinds of information.

[0097] Exemplarily, the interface memory unit 143 may determine the similarity between the current interface fusion understanding information and each interface fusion memory information, and determine the interface fusion memory information with the largest similarity greater than a threshold as the interface fusion memory information similar to the current interface fusion understanding information.

[0098] It should be noted that if the interface memory unit 143 does not find interface fusion memory information similar to the current interface fusion understanding information, the interface fusion understanding information of the current display interface is sent to the server 200 for processing, so that the server 200 directly generates a control instruction sequence based on the interface fusion understanding information.

[0099] Instruction execution module 150 , used to receive the instruction sequence and execute the control instructions in the instruction sequence in sequence so that the terminal device can realize a specific function. For example, the control instructions are executed in sequence so that the terminal device automatically fills in Lucy's passport information.

[0100] The server 200 , also called a cloud server or a cloud side, includes a structured memory module 210 , a memory database 220 , and a memory reasoning framework module 230 .

[0101] Structured Memory Module 210 , used to obtain, process and store the user's memory information after obtaining the user's authorization. Exemplarily, the structured memory module 210 includes a memory acquisition unit 211, a memory understanding unit 212 and a memory storage unit 213.

[0102] Memory acquisition unit 211 Used to obtain user memory, that is, the user's usage information of the terminal device, referred to as user usage information.

[0103] In some embodiments, the user usage information includes the conversation information between the user and the voice assistant, such as "Xiaoyi, I like to drink milk tea from store A, but not coffee", "Please send a text message to Lucy to say happy birthday", etc.

[0104] In other embodiments, user usage information includes information that the user actively gives to the voice assistant for storage, which may be a phone number, card information (such as ID card information, passport information), pictures, web pages, videos, text, files, etc. For example, the user can give the information to the voice assistant for storage through a voice control command "help me save Lucy's ID card information in the display interface". Alternatively, the user can drag the target information to be stored to the information storage entrance of the voice assistant through a drag operation, thereby giving the information to the voice assistant for storage.

[0105] In some other embodiments, the user usage information includes factual content recorded based on the user's long-term (e.g., more than 7 days) usage habits of the voice assistant, such as telephone number, card information (such as ID card information, passport information), event information, important day information (such as birthdays, anniversaries, meeting days), etc.

[0106] In some other embodiments, the user usage information includes information stored by the user in the terminal device, such as information stored by the user in a memo, photos, files, audio, video, web links, etc. stored by the user.

[0107] It should be noted that the memory acquisition unit 211 can acquire user information according to user instructions, or can acquire user information by active memory. Active memory means that after the user authorizes, the terminal device actively explores the user's memory, for example, by extracting a large memory model or using active memory technology to acquire memo information, conversation information between the user and the voice assistant, and send it to the memory understanding unit 212.

[0108] Memory Comprehension Unit 212 , used to memorize and understand user information and convert it into structured user memory information. Structured user memory information is used to abstract and summarize user information, which includes user memory classification, memory attributes (according to the attributes specific to the classification), memory content and other information (see Table 1). It can be understood that the terminal device can reverse the original memory content based on the structured information, and it is convenient for the terminal device to query, understand and display it later. Structured user memory information can be stored in the terminal device for a long time, which is convenient for the terminal device to query and match information later.

[0109] Table 1

[0110]

[0111] Taking a passport as an example, Table 2 shows structured user memory information about a passport.

[0112] Table 2

[0113]

[0114] Referring to Example 1 and Example 2 below, the memory understanding unit 212 can generate corresponding structured user memory information based on different user information (eg, user instructions).

[0115] Example 1:

[0116] User instruction: Write down that my daughter’s birthday is September 1st every year.

[0117] Structured user memory information: [{"subCategory":"Birthday","importantDayType":"Daughter's birthday","importantDate":"September 1st of every year","subject":"daughter"}]}.

[0118] Example 2:

[0119] User instruction: Please write down my daughter’s ID number is 412728********3789.

[0120] Structured user memory information: [{"subCategory":"ID card","subject":"daughter","cardNumber":"412728********3789"}].

[0121] Memory storage unit 213 , used to encrypt and store structured user memory information for subsequent use with user authorization. For example, encrypt and store the structured user memory information in the above examples 1 and 2.

[0122] Memory database 220 , used to store the user's long-term memory information and short-term memory information. Among them, the long-term memory information includes the encrypted user memory information (such as the mask and ID of the user memory information). The short-term memory information includes the user's operation record information on the terminal device, etc.

[0123] It should be noted that the server 200 can synchronize the user's long-term memory information across devices so that all electronic devices logged in to the same user account (such as a Huawei account) can use the user's long-term memory information.

[0124] The memory reasoning framework module 230 is used to generate a control instruction sequence according to user instructions and interface information.

[0125] In some embodiments, the memory reasoning framework module 230 receives the interface fusion understanding information (including user instructions and interface information) of the current display interface, but does not receive the historical layout fusion memory information. In this case, the memory reasoning framework module 230 directly uses the reasoning model (such as a large language model (LLM)) to determine the corresponding control instruction sequence, and sends the control instruction sequence to the instruction execution module 150 of the terminal device 100.

[0126] In other embodiments, the memory reasoning framework module 230 not only receives the interface fusion understanding information of the current display interface, but also receives the interface fusion memory information similar to the interface fusion understanding information. Based on this, the memory reasoning framework module 230 refers to the interface fusion memory information (such as the control instruction sequence therein), generates a control instruction sequence corresponding to the current interface fusion understanding information, and sends the control instruction sequence to the instruction execution module 150 of the terminal device 100.

[0127] It should be noted that the control instruction sequence in this embodiment includes multiple atomic control instructions that are executed sequentially. Among them, the atomic control instruction refers to one or a series of instructions that cannot be interrupted during the running process. Exemplarily, the atomic control instruction includes an application programming interface (API), a search operation, a click operation, a sliding operation, a dialogue operation, a confirmation operation, etc. This embodiment does not limit the specific content of the control instruction.

[0128] In addition, the memory reasoning framework module 230 will generate interface fusion memory information based on the interface fusion understanding information and control instruction sequence of the current display interface, and send it to the interface memory unit 143 of the terminal device 100 for storage for subsequent use.

[0129] After introducing the device control system, the device control method provided in this embodiment is specifically described below.

[0130] In the device control method, see Figure 4 As shown in S401 to S402 in the figure, first, the terminal device obtains a first user instruction, which is used to instruct the terminal device to perform a target operation. Exemplarily, the target information may be filling in information, purchasing goods, etc. Subsequently, in response to the first user instruction, the terminal device automatically performs the target operation according to the interface information of its display interface and the first target information. The first target information is obtained by the terminal device from the user memory information stored therein according to the target operation.

[0131] Through the method provided in this embodiment, after receiving the first user instruction, the terminal device can combine the interface information and the user memory information to implement the control operation that meets the actual needs of the user and accurately execute the user instruction.

[0132] It can be understood that in the device control method provided in this embodiment, the terminal device must first store the user memory information, and then execute the first user instruction according to the user memory information and the interface information. The implementation process of these two parts is specifically described below in combination with user operations.

[0133] (I) Storing user memory information

[0134] In this embodiment, the terminal device may store the user's memory information according to the user's information storage instruction, or may actively explore the user's memory information. This embodiment does not limit this.

[0135] In a scenario where a terminal device stores user memory information according to the user's information storage instructions, the information storage instructions include: voice control instructions, text control instructions, or touch operations in a display interface input by the user through a voice assistant.

[0136] See also Figure 5 As shown, after the terminal device receives the user's information storage instruction through the instruction acquisition module 110, it first sends it to the instruction parsing module 120 for user intention parsing to determine the information to be stored. Subsequently, the instruction parsing module 120 sends the information to be stored to the structured memory module 210 of the server 200 for processing. The structured memory module 210 processes the information into structured user memory information and then returns it to the terminal device, which stores it in the user memory module 130.

[0137] In some implementations, the information storage instruction carries information to be stored. After receiving the information storage instruction, the terminal device converts the information carried in the instruction into structured user memory information and stores it locally.

[0138] For example Fig. 6A As shown, the user sends a voice control command to the voice assistant (such as Xiaoyi) "Xiaoyi Xiaoyi, help me store my daughter's birthday, September 1st". Correspondingly, the terminal device receives the voice control command through the command acquisition module 110, and then sends it to the command parsing module 120 for user intention analysis. The command parsing module 120 sends the information to be stored "My daughter's birthday is September 1st" to the structured memory module 210 of the server 200 for processing. The structured memory module 210 processes the information into structured user memory information and then returns it to the terminal device, which stores it in the user memory module 130.

[0139] For example Figure 6B As shown, the user sends a text control command to the voice assistant (such as Xiaoyi) through the dialog box of the voice assistant, "Help me store my daughter's birthday, September 1st." The terminal device converts "My daughter's birthday, September 1st" into structured user memory information and stores it in the terminal device.

[0140] Another example Figure 6C As shown, the display interface of the electronic device displays information to be stored, such as Zhang San's information. Based on this, in response to the user's drag operation on the information to be stored, the terminal device will display a storage control, such as an "AI safekeeping" control. After the terminal device detects that the user drags the information to be stored to the storage control and releases it, the terminal device processes Zhang San's information into structured user memory information and stores it in the terminal device.

[0141] In other implementations, the information storage instruction does not carry the information to be stored, but it can instruct the terminal device to store the information to be stored in the current display interface (such as the second interface). Based on this, the terminal device stores the target information in the second interface as user memory information according to the information storage instruction.

[0142] For example Fig.6D As shown, the display interface of the electronic device displays information to be stored, such as Zhang San’s ID card information. Based on this, the user sends a voice control command to the voice assistant (such as Xiaoyi) "Xiaoyi Xiaoyi, help me store Zhang San’s information." The terminal device receives the voice control command through the command acquisition module 110, and then sends it to the command parsing module 120 for user intent analysis. The command parsing module 120 processes Zhang San’s information to be stored into structured memory information through the structured memory module 210 of the server 200, and stores it in the user memory module 130.

[0143] In the scenario where the terminal device obtains user authorization and actively explores the user's memory information, the terminal device can use, for example, a memory extraction large model, active memory technology, etc. to convert the information stored in the terminal device or the factual content involved in the user's usage habits into user memory information storage. It can be understood that in the process of the terminal device actively exploring the user's memory information, usually no user operation is required.

[0144] (ii) Executing the first user instruction according to the user memory information and the interface information

[0145] Figure 7 is a schematic flow chart of a device control method provided by an embodiment of the present application. Figure 7 As shown, the method involves a process in which a terminal device executes a complex user instruction, and specifically includes the following steps S701 to S707.

[0146] S701: The terminal device obtains a first user instruction, where the first user instruction is used to instruct the terminal device to perform a target operation.

[0147] The terminal device can obtain the first user instruction through the instruction acquisition module 110. In this embodiment, the terminal device supports obtaining diversified user instructions. For example, the first user instruction can be a voice control instruction or a text control instruction obtained through a voice assistant, or a user's selection operation on an application control in the terminal device, or a pressing operation on a physical button of the terminal device, etc. This embodiment does not specifically limit its type.

[0148] Since the first user instruction is issued by the user according to his actual control needs, based on the diversity of user needs, the target operation that the first user instruction instructs the terminal device to perform is also diverse. Exemplarily, the target operation is to control the terminal device to fill in information (such as passport information, ID card information, etc.) on the display interface; or, the target operation is to control the terminal device to purchase goods (such as daily necessities, food, takeout, books, etc.) through an application, where purchasing goods can be understood as generating a purchase order for the goods.

[0149] S702: The terminal device determines whether the first user instruction is a second-category user instruction.

[0150] After receiving the first user instruction through the instruction acquisition module 110, the terminal device sends it to the instruction parsing module 120 for instruction parsing to determine the instruction type of the first user instruction. If the first user instruction belongs to the first type of user instruction (i.e., simple user instruction), the terminal device directly generates and executes the corresponding control operation according to the result of the first user instruction to realize the user's intention. If the first user instruction is a second type of user instruction (i.e., complex user instruction), the terminal device executes subsequent steps S703 to S707.

[0151] S703: If the first user instruction is a second-category user instruction, the terminal device determines the interface information of the current display interface.

[0152] In this embodiment, the first user instruction may not be clear, such as "please fill out the form for me" or "please order me some milk tea takeout". Therefore, the terminal device needs to understand the current display interface, and then combine the user instruction and the understood interface information to determine the specific content to be filled in.

[0153] In S703, illustratively, after determining that the first user instruction is a second-category user instruction, the terminal device determines its interface information through the interface fusion understanding module 140, and the interface information includes content information and layout information of the display interface.

[0154] In some embodiments, the interface fusion understanding module 140 may also search its interface memory unit 143 for interface fusion memory information similar to the first user instruction and interface information.

[0155] S704: The terminal device sends the first user instruction and interface information to the server.

[0156] In some embodiments, if the interface fusion understanding module 140 of the terminal device also searches for interface fusion memory information similar to the first user instruction and interface information, the terminal device sends the interface fusion memory information together with the first user instruction and interface information to the server.

[0157] S705: The server generates a control instruction sequence according to the first user instruction and the interface information.

[0158] The memory reasoning framework module 230 of the server can generate a control instruction sequence according to the received information. The specific process of generating the control instruction sequence can be found in the functional introduction of the memory reasoning framework module 230 above, which will not be described here.

[0159] Taking the scenario of filling in passenger information as an example, the control instruction sequence generated by the memory reasoning framework module 230 of the server includes multiple atomic control instructions executed sequentially. As shown in Table 3, the control instruction sequence includes control instruction 1, control instruction 2 and control instruction 3. Among them, control instruction 1 is used to control the terminal device to fill in the passenger's name on the display interface, control instruction 2 is used to control the terminal device to fill in the passenger's ID number on the display interface, and control instruction 3 is used to control the terminal device to fill in the passenger's phone number on the display interface.

[0160] Table 3 Control instruction sequence

[0161] Command name Function Control instruction 1 Control to fill in the name of the passenger Control Instructions 2 Control the filling of the passenger's ID number Control Instructions 3 Control to fill in the passenger's phone number

[0162] S706: The server sends a control instruction sequence to the terminal device.

[0163] S707, the terminal device obtains the first target information from the user memory information according to the control instruction sequence, and executes the control instruction sequence according to the first target information to achieve the target operation.

[0164] It should be noted that since there is no user memory information in the server, the control instruction sequence it returns does not include the first target information required for instruction execution. The terminal device needs to obtain the first target information from the user memory information stored in it, and then complete the corresponding control function according to the control instruction sequence.

[0165] In some embodiments, the target operation indicated by the first user instruction is: filling in information on the display interface. Then, the terminal device obtains the first target information as information to be filled in, such as name, ID number, telephone number, etc.

[0166] In some other embodiments, the target operation indicated by the first user instruction is: purchasing a target product. Then, the terminal device obtains the first target information as product information of the target product, such as product name, quantity, product attributes, application program identifier, and delivery address.

[0167] Taking the control execution sequence shown in Table 3 as an example, the instruction execution module 150 obtains the corresponding user information from the user memory module 130 according to the control instruction sequence, and fills it in the corresponding filling position of the display interface. For example, according to the user instruction sequence 2, the ID card number "610******8222" of Zhang San is obtained from the user memory module 130, and it is filled in the filling slot of the ID card number. Or, according to the user instruction sequence 3, the telephone number "135***2195" of Zhang San is obtained from the user memory module 130, and it is filled in the filling slot of the telephone number.

[0168] In addition, after successfully executing the target operation, the terminal device can store the interface information of the current display interface, the first user instruction and the control instruction sequence as interface fusion memory information in the interface memory unit 143 for reference when the terminal device subsequently processes similar user instructions, so as to improve the accuracy of the control instruction sequence.

[0169] Through the device control method provided in this embodiment, after receiving complex user instructions, the terminal device can combine interface analysis (including content analysis and layout analysis) technology to understand the current interface information, and combine user memory information to dynamically generate a clear and executable control plan, thereby realizing continuous control operations that meet the actual needs of the user and accurately executing user instructions.

[0170] Based on the above description, it can be known that the first user instruction can control the terminal device to perform various target operations, such as controlling the terminal device to automatically fill in information on the display interface, and controlling the terminal device to automatically purchase the target product in the application. The device control method provided in the embodiment of the present application is specifically described below in combination with these two examples.

[0171] (1) Automatically fill in information on the display interface

[0172] This embodiment supports diversified user commands, such as voice control commands, text control commands, or touch control operations, etc. The following combines different forms of user commands, taking the example of a user adding passenger Zhang San's information in the "Add Passenger" interface of the ticket purchase application, to exemplify the process of automatically filling in information on the display interface of the terminal device.

[0173] In some embodiments, the terminal device may first display an interface for information to be filled in, such as a first interface, wherein the first interface includes a target area for information to be filled in. Subsequently, the terminal device automatically fills in the first target information in the target area in response to the first user instruction and according to the interface information of the first interface. The first target information is determined from the memory information stored in the terminal device according to the first user instruction and the interface information of the first interface.

[0174] Example 1: Automatically fill in information on the display interface based on voice control instructions.

[0175] See also Figure 8 As shown in (a) to (c) in the figure, first, the terminal device displays the "Add Passenger" interface according to the user's operation, and the information to be filled in the interface includes "Name", "ID Number", and "Phone Number". Subsequently, in response to the voice control command "Xiaoyi Xiaoyi, help me fill in Zhang San's information" input by the user through the voice assistant "Xiaoyi", the terminal device automatically fills in the name "Zhang San", the ID number "610******8222" and the phone number "135***2195".

[0176] Optional, see Figure 8 As shown in (b), during the process of automatically filling in information, the terminal device can display a prompt message, such as "helping you fill in the form...", to prompt the user that the terminal device is processing.

[0177] Example 2: Automatically fill in information on the display interface according to text control instructions.

[0178] See also Fig. 9 As shown in (a) to (c) in the figure, first, the terminal device displays the "Add Passenger" interface according to the user's operation, and the information to be filled in the interface includes "name", "ID number", and "telephone number". Subsequently, in response to the text control instruction "Help me fill in Zhang San's information" entered by the user in the dialog box of the voice assistant "Xiaoyi", the terminal device automatically fills in the name "Zhang San", the ID number "610******8222" and the telephone number "135***2195" in the "Add Passenger" interface.

[0179] Example 3: Automatically fill in information on the display interface according to the selection operation of the first control.

[0180] In this embodiment, the terminal device first fills in the index words on the first interface according to the user's input operation. Exemplarily, the index words include: person names, event names, place names, keywords, product names, dates, etc. After detecting the index words, the terminal device displays a first control on the first interface according to the index words. The first control is used to indicate that the terminal device stores the first target information corresponding to the index words. In response to the selection operation of the first control, the first target information is automatically filled in the target area.

[0181] See also Fig.10 As shown in (a) to (d), first, the terminal device displays the "Add Passenger" interface according to the user operation, and the information to be filled in the interface includes "Name", "ID Number", and "Phone Number". Subsequently, the terminal device fills in the index word "Zhang San" in the name item through the input method application according to the user's input operation. After detecting the index word "Zhang San", the terminal device displays a first control for obtaining relevant information about Zhang San on the interface, such as an "Auto Fill" control. In response to the user's selection operation of the first control "Auto Fill", the terminal device automatically fills in the name "Zhang San", ID number "610******8222" and phone number "135***2195" in the "Add Passenger" interface.

[0182] See also Fig.11 As shown in (a) to (c) in the figure, first, the terminal device displays the "Add Passenger" interface according to the user operation, and the information to be filled in the interface includes "Name", "ID Number", and "Phone Number". Subsequently, the terminal device searches for the index term "Zhang San" in the input method interface according to the user's input operation, and determines the search term as the first control. In response to the user's selection operation of the first control "Zhang San", the terminal device automatically fills in the name "Zhang San", the ID number "610******8222" and the phone number "135***2195" in the "Add Passenger" interface.

[0183] In some other embodiments, the terminal device first receives the first user instruction, and then displays the first interface according to the first user instruction, wherein the first interface includes a target area for information to be filled in. Finally, the terminal device automatically fills in the first target information in the target area according to the interface information of the first interface. For example, the first user instruction may be "help me fill in Form A on the desktop".

[0184] It can be seen that through the device control method provided in this embodiment, the terminal device can realize "one-click automatic form filling" by perceiving and understanding the display interface and combining the user's memory information, which has a good user experience.

[0185] The following is a detailed description of other possible control operations involved in the "one-click automatic form filling" process.

[0186] After the terminal device receives the first user instruction to fill in information, if the terminal device finds through analysis that the first user instruction is unclear or the user's memory information is missing, resulting in the terminal device being unable to generate a control instruction sequence, then the terminal device can further question the user to obtain relevant information.

[0187] In some embodiments, the terminal device receives a first user instruction to fill in information, such as "please help me fill in Zhang San's information", but cannot determine the area where the information is to be filled in. For example, the interface currently displayed by the terminal device is the desktop, and there is Form A on the desktop but Form A is not opened, that is, the current display interface of the terminal device does not include the target area where the information is to be filled in. Based on this, the terminal device can further inquire about the filling location of the information, for example, the terminal device can ask the user "Where do you want to fill it in?" Assuming that the user's answer is "Fill in Form A on the desktop", the terminal device opens Form A on the desktop and then fills in Zhang San's information in Form A.

[0188] In other embodiments, after receiving the first user instruction to fill in the information, the terminal device finds that the information to be filled in is missing in the user's memory information, so the terminal device can ask the user for the missing information. For example, in the scenario where the user instructs the terminal device to fill in Zhang San's information, it is detected that all or part of Zhang San's information is missing in the user's memory information, so the terminal device can ask the user through the voice assistant "What is Zhang San's ID number and phone number?" After receiving the missing information supplemented by the user, the terminal device can fill in Zhang San's information in the current display interface.

[0189] In order to ensure the accuracy of the user information filled in, after receiving the control instruction list and determining the user information to be filled in, the terminal device can display the user information as candidate information in the display interface for the user to further confirm or modify. Subsequently, in response to the user's selection operation of the first target information in the candidate information, the terminal device fills the first target information in the corresponding position of the display interface.

[0190] See also Fig.12As shown, taking the example of a user adding passenger Zhang San's information in the "Add Passenger" interface of the ticket purchase application, the terminal device can first display a prompt box, which includes candidate information, a selection control for each candidate information, a cancel control, and a confirmation control. Exemplarily, the candidate information includes the name "Zhang San", "mobile phone number 135****2195", and "ID number 610******8222". The selection control is used to control whether to fill in the corresponding candidate information. The cancel control is used to cancel the filling of user information in the current interface. The confirmation control can be the "Start Filling" control in the figure, which is used to control the terminal device to fill in the candidate information selected by the user. In response to the user's selection operation on the confirmation control, the terminal device confirms the candidate information selected by the user as the first target information, and fills it in the corresponding slot of the current display interface.

[0191] In addition, when the user is controlling the terminal device to fill in information, the user may need to pause or re-fill in the user information due to command errors, changes in filling plans, etc. In this embodiment, the terminal device can also respond to the user's need to pause or re-fill in the user information, as shown below.

[0192] In some embodiments, if the terminal device receives a pause instruction during the process of automatically filling in the first target information, such as "Xiaoyi Xiaoyi pause filling" or "Xiaoyi Xiaoyi, don't fill it in yet", then in response to the pause instruction, the terminal device pauses filling in the first target information. For example, when the terminal device is filling in Zhang San's information on the "Add Passenger" interface, if a pause instruction is received, then the filling in of Zhang San's information is paused.

[0193] In some other embodiments, after the terminal device pauses filling in the first target information or after the first target information is filled in, the terminal device receives a second user instruction. Then, in response to the second user instruction, the terminal device automatically replaces the filled in first target information with the second target information. The second target information is determined from the user memory information stored in the terminal device according to the second user instruction and the interface information of the first interface.

[0194] For example, see Fig.13 As shown in (a) to (c), after the terminal device fills in Zhang San's information according to the first user instruction, the terminal device receives the second user instruction "Help me re-fill in Li Si's information". Then, the terminal device deletes Zhang San's information in the display interface, and obtains Li Si's information from the user memory information stored in the terminal device according to the received control instruction sequence, and automatically fills it in the information slot corresponding to the current display interface. Exemplarily, Li Si's information includes: name "Li Si", ID number "610******0335" and phone number "150****6930".

[0195] Optionally, the terminal device may also update part of the filled-in first target information according to a user instruction, for example, updating Zhang San's mobile phone number from "135****2195" to "135****6789".

[0196] It should be noted that the information processing of the terminal device filling in the second target information in response to the second user instruction is the same as the processing process of filling in the second target information in response to the first user instruction. For details, please refer to steps S701 to S707 shown above, which will not be repeated here.

[0197] (2) Automatically purchase products

[0198] In combination with different forms of user instructions, the following uses the example of a user instructing a terminal device to "order milk tea takeout" to exemplify the process of a terminal device automatically purchasing goods according to the user instruction.

[0199] Example 1: Purchasing goods based on voice control instructions.

[0200] In response to the voice control command "Order me a takeaway milk tea" input by the user through the voice assistant "Xiaoyi", the voice assistant replies to the user through voice that it will place an order based on the user's daily habits, for example, "OK, I will help you place an order based on your daily habits. You usually like to use App A to place orders, so we will use App A to help you place an order." Subsequently, the terminal device automatically executes operations such as opening App A, entering the online store, selecting the target milk tea, and selecting the attributes of the milk tea according to the user's daily habits, and finally generates a milk tea purchase order.

[0201] Example 2: Purchase goods according to text control instructions.

[0202] See also Fig.14 As shown, in response to the text control instruction "Order me a takeaway milk tea" input by the user through the dialog box of the voice assistant "Xiaoyi", the voice assistant replies to the user in the dialog box that it will place an order according to the user's daily habits, for example, "OK, I will help you place an order according to your daily habits. You usually like to use application A to place an order, so we will use application A to help you place an order." Subsequently, the terminal device opens the takeaway application, enters the online store, selects the target milk tea, and finally generates a milk tea purchase order according to the user's daily habits.

[0203] After the terminal device generates a purchase order for a product, if the user has authorized the application to make a password-free payment, the application will automatically pay the order without a password. If the user has not authorized the application to make a password-free payment, the user needs to manually control the payment order.

[0204] It should be noted that the terminal device usually displays multiple levels of display interfaces during the purchase of goods. The terminal device needs to integrate and understand each level of display interface before determining the control instruction sequence corresponding to the interface. Therefore, after each new display interface is displayed, the terminal device needs to determine the interface information through the interface integration and understanding module 140, and send the interface information to the memory reasoning framework module 230 of the server, so that the memory reasoning framework module 230 can determine the control instruction sequence under the newly displayed interface based on the interface information and user instructions.

[0205] It can be understood that the terminal device can successfully generate a product purchase order by repeatedly executing the above-mentioned process of "interface understanding - executing the current interface control instruction sequence - displaying the new interface" for multiple times.

[0206] For example Fig.15 As shown, when the terminal device receives the user's text control instruction "Order me a takeaway milk tea", the terminal device first generates a first control instruction sequence according to the text control instruction and the interface information of the desktop, including: a control instruction to open the takeaway application A. After the terminal device executes the control instruction to open the takeaway application A, the store selection interface of the takeaway application A is displayed, including multiple stores such as store 1, store 2, and store 3. Subsequently, the terminal device generates a second control instruction sequence according to the interface information of the store selection interface, including: a control instruction to open store 1. After the terminal device executes the control instruction to open store 1, the product selection interface of store 1 is displayed, including multiple products such as product A, product B, and product C. Then, the terminal device generates a third control instruction sequence according to the interface information of the product selection interface, including: a control instruction to select product B. After the terminal device executes the control instruction to select product B, the attribute selection interface of product B is displayed. Exemplarily, the attribute includes the cup type, temperature, sweetness, etc. of milk tea. Next, the terminal device generates a fourth control instruction sequence according to the interface information of the attribute selection interface, including: a cup type selection instruction, a temperature selection instruction, and a sweetness selection instruction. The terminal device executes each selection instruction in turn to determine the attributes of the product. Finally, the terminal device generates and executes the product purchase instruction according to the product attributes to generate a product purchase order.

[0207] It can be seen that in the above process, the terminal device can perform personalized control operations according to user preferences, habits or executions, and has a better user experience.

[0208] In order to further improve the intelligent execution effect of user instructions, the terminal device can optimize the generated atomic control instruction sequence based on user feedback information and full-scene control perception results. Among them, user feedback information includes but is not limited to user scoring, evaluation, likes / dislikes, etc., and the full-scene control perception results include the fusion perception results of control changes and interface information. Exemplarily, if a control behavior fails or does not proceed as expected, the control result will be stored in the interface memory unit 143, and the machine learning algorithm can be used to learn the error when the control execution is generated next time, and avoid similar problems in the future.

[0209] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0210] An embodiment of the present application also provides an electronic device, which includes one or more processors and a memory; the memory is coupled to the one or more processors; the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the device control method shown in the above-mentioned embodiments.

[0211] Fig.16 1 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may include a processor 1610, an external memory interface 1620, an internal memory 1621, a universal serial bus (USB) interface 1630, a charging management module 1640, a power management module 1641, a battery 1642, an antenna 1, an antenna 2, a mobile communication module 1650, a wireless communication module 1660, an audio module 1670, a speaker 1670A, a receiver 1670B, a microphone 1670C, an earphone interface 1670D, a sensor module 1680, a button 1690, a motor 1691, an indicator 1692, a camera 1693, a display screen 1694, and a subscriber identification module (SIM) card interface 1695, etc.

[0212] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. As an example, when the electronic device is a mobile phone or a tablet computer, it may include all the components shown in the figure, or it may include only some of the components shown in the figure.

[0213] The processor 1610 may include one or more processing units, for example, the processor 1610 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0214] The charging management module 1640 is used to receive charging input from a charger, which may be a wireless charger or a wired charger.

[0215] The power management module 1641 is used to connect the battery 1642, the charging management module 1640 and the processor 1610. The power management module 1641 receives input from the battery 1642 and / or the charging management module 1640, and provides power to the processor 1610, the internal memory 1621, the external memory, the display screen 1694, the camera 1693, and the wireless communication module 1660.

[0216] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 1650, wireless communication module 1660, a modem processor, and a baseband processor.

[0217] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.

[0218] The mobile communication module 1650 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 1650 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 1650 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 1650 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0219] The wireless communication module 1660 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to electronic devices. The wireless communication module 1660 can be one or more devices integrating at least one communication processing module. The wireless communication module 1660 receives electromagnetic waves via antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 1610. The wireless communication module 1660 can also receive the signal to be sent from the processor 1610, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through antenna 2.

[0220] The electronic device implements the display function through a GPU, a display screen 1694, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1694 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1610 may include one or more GPUs that execute program instructions to generate or change display information.

[0221] The display screen 1694 is used to display images, videos, etc. The display screen 1694 includes a display panel. In some embodiments, the electronic device may include 1 or N display screens 1694, where N is a positive integer greater than 1.

[0222] The electronic device can realize the shooting function through ISP, camera 1693, video codec, GPU, display screen 1694 and application processor.

[0223] The ISP is used to process the data fed back by the camera 1693. For example, when taking a photo, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The camera 1693 is used to capture static images or videos.

[0224] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0225] Video codecs are used to compress or decompress digital video.

[0226] The external memory interface 1620 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 1610 through the external memory interface 1620 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0227] The internal memory 1621 may be used to store computer executable program codes, which may include instructions. The processor 1610 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 1621. The internal memory 1621 may include a program storage area and a data storage area.

[0228] In addition, the internal memory 1621 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0229] The electronic device can implement audio functions through the audio module 1670, the speaker 1670A, the receiver 1670B, the microphone 1670C, the headphone jack 1670D, and the application processor.

[0230] The audio module 1670 is used to convert digital audio signals into analog audio signals for output, and also to convert analog audio inputs into digital audio signals. The audio module 1670 can also be used to encode and decode audio signals.

[0231] Speaker 1670A, also called "speaker", is used to convert audio electrical signals into sound signals. The electronic device can listen to music or listen to hands-free calls through speaker 1670A. For example, the speaker can play the comparison analysis results provided in the embodiment of the present application.

[0232] The receiver 1670B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the voice can be received by placing the receiver 1670B close to the human ear.

[0233] Microphone 1670C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by approaching the microphone 1670C with his mouth to input the sound signal into the microphone 1670C.

[0234] The headphone jack 1670D is used to connect wired headphones.

[0235] The sensor module 1680 may include a pressure sensor 1680A, a gyroscope sensor 1680B, an air pressure sensor 1680C, a magnetic sensor 1680D, an acceleration sensor 1680E, a distance sensor 1680F, a proximity light sensor 1680G, a fingerprint sensor 1680H, a temperature sensor 1680J, a touch sensor 1680K, an ambient light sensor 1680L, a bone conduction sensor 1680M, and the like.

[0236] The key 1690 includes a power key, a volume key, etc. The key 1690 may be a mechanical key or a touch key. The electronic device may receive key input and generate key signal input related to user settings and function control of the electronic device.

[0237] Motor 1691 can generate vibration prompts. Motor 1691 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0238] Indicator 1692 can be an indicator light, which can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0239] The SIM card interface 1695 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device by inserting into or removing from the SIM card interface 1695.

[0240] The present application also provides a chip, see Fig.17 As shown, the chip includes a processor and a memory, in which a computer program is stored. When the computer program is executed by the processor, the device control method in the above embodiments is implemented.

[0241] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the device control method provided in the above embodiments is implemented.

[0242] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device implements the device control method provided in the above embodiments.

[0243] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0244] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0245] In the embodiments provided in the present application, the division of each framework or module is only a logical function division. There may be other division methods in actual implementation. For example, multiple frameworks or modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0246] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0247] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0248] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0249] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A device control method, characterized in that: Applied to a terminal device, the method comprises: Displaying a first interface, wherein the first interface includes a target area for information to be filled in; In response to a first user instruction, first target information is automatically filled in the target area; wherein the first target information is memory information associated with the first user instruction and the interface information of the first interface.

2. The method according to claim 1, characterized in that The first user instruction includes: Voice control commands; or, Text control instructions; or, The selection operation of the first control is performed, wherein the first control is displayed according to the index word input in the target area, and the index word is associated with the first target information.

3. The method according to claim 1 or 2, characterized in that: The first user instruction includes a selection operation on a first control, and the automatically filling in the first target information in the target area in response to the first user instruction includes: Displaying index words on the input method interface according to input operations in the input method application; Determine the index word as the first control; In response to a selection operation on the first control, the first target information is automatically filled in the target area.

4. The method according to claim 1 or 2, characterized in that: The first user instruction includes a selection operation on a first control, and the automatically filling in the first target information in the target area in response to the first user instruction includes: According to the input operation, fill in the index word in the target area of ​​the first interface; Displaying a first control on the first interface according to the index word; In response to a selection operation on the first control, the first target information is automatically filled in the target area.

5. The method according to any one of claims 1 to 4, characterized in that: In response to the first user instruction, automatically filling in the first target information in the target area includes: In response to a first user instruction, displaying candidate information on the first interface, the candidate information being determined from memory information stored in the terminal device according to the first user instruction and interface information of the first interface; determining the first target information from the candidate information according to a control operation; The first target information is automatically filled in the target area.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: During the process of automatically filling in the first target information in the target area, a pause instruction is received; In response to the pause instruction, the terminal device pauses filling in the first target information.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: After pausing to fill in the first target information, or after completing filling in the first target information, receiving a second user instruction; In response to the second user instruction, automatically replacing the first target information that has been filled in with the second target information; The second target information is determined from the memory information stored in the terminal device according to the second user instruction and the interface information of the first interface.

8. The method according to any one of claims 1 to 7, characterized in that: Before automatically filling in the first target information in the target area in response to the first user instruction, the method further includes: receiving an information storage instruction, wherein the information storage instruction carries the first target information; The first target information is stored.

9. The method according to any one of claims 1 to 7, characterized in that: Before automatically filling in the first target information in the target area in response to the first user instruction, the method further includes: displaying a second interface, wherein the second interface includes the first target information; receiving an information storage instruction, wherein the information storage instruction is used to instruct the terminal device to store the first target information; The first target information is stored in the memory information.

10. A device control method, characterized in that: Applied to a terminal device, the method comprises: Acquire a first user instruction, where the first user instruction is used to control the terminal device to perform a target operation; In response to the first user instruction, the target operation is automatically performed according to the interface information and first target information of the terminal device, wherein the first target information is memory information associated with the first user instruction and the interface information.

11. The method according to claim 10, characterized in that The target operation includes purchasing a target commodity, and the first target information includes commodity information of the target commodity; and the automatically executing the target operation in response to the first user instruction according to the interface information of the terminal device and the first target information includes: In response to the first user instruction, a purchase order for the target product is generated according to the interface information of the terminal device and the product information of the target product.

12. The method according to claim 11, characterized in that The product information includes: product name, quantity, product attributes, application program identifier and delivery address.

13. The method according to claim 10, characterized in that The target operation includes filling in information, and the first target information includes content to be filled in.

14. The method according to claim 13, characterized in that Before obtaining the first user instruction, the method further includes: displaying a first interface, the first interface including a target area for information to be filled in; and The automatically executing the target operation in response to the first user instruction according to the interface information of the terminal device and the first target information includes: in response to the first user instruction, automatically filling in the first target information in the target area according to the interface information of the terminal device, wherein the first target information is determined from the memory information stored in the terminal device according to the first user instruction and the interface information of the first interface.

15. The method according to claim 13, characterized in that The step of automatically executing the target operation in response to the first user instruction and according to the interface information of the terminal device and the first target information includes: In response to the first user instruction, displaying a first interface, wherein the first interface includes a target area for information to be filled in; The first target information is automatically filled in the target area according to the interface information of the first interface, wherein the first target information is determined from the memory information stored in the terminal device according to the first user instruction and the interface information of the first interface.

16. The method according to claim 14 or 15, characterized in that The automatically filling in the first target information in the target area includes: Displaying candidate information on the first interface, wherein the candidate information is determined from memory information stored in the terminal device according to the first user instruction and interface information of the first interface; determining the first target information from the candidate information according to a control operation; The first target information is automatically filled in the target area.

17. The method according to any one of claims 13 to 16, characterized in that: The method further comprises: After pausing to fill in the first target information, or after completing filling in the first target information, receiving a second user instruction; In response to the second user instruction, automatically replacing the first target information that has been filled in with the second target information; The second target information is determined from the memory information stored in the terminal device according to the second user instruction and the interface information of the first interface.

18. The method according to any one of claims 10 to 17, characterized in that: The method further comprises: In the process of automatically executing the target operation according to the first target information and the interface information of the terminal device, a pause instruction is received; In response to the pause instruction, execution of the target operation is paused.

19. The method according to any one of claims 10 to 18, characterized in that: Before receiving the first user instruction, the method further includes: receiving an information storage instruction, wherein the information storage instruction carries the first target information; The first target information is stored.

20. The method according to any one of claims 10 to 19, characterized in that: Before receiving the first user instruction, the method further includes: displaying a second interface, wherein the second interface includes the first target information; receiving an information storage instruction, wherein the information storage instruction is used to instruct to store the first target information; The first target information is stored.

21. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors; The memory is used to store computer program codes, where the computer program codes include computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 9 or any one of 10 to 20.

22. A chip, characterized in that: The chip is applied to an electronic device, and the chip includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 9 or any one of 10 to 20.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 or any one of 10 to 20 is implemented.

Images