Device control method, electronic device, readable storage medium and chip

Abstract

The present application relates to the technical field of artificial intelligence. Provided are a device control method, an electronic device, a readable storage medium and a chip. The method comprises: displaying a first interface, wherein the first interface comprises a target area in which information is to be filled in; and in response to a first user instruction, automatically filling first target information in the target area, wherein the first target information is determined on the basis of the first user instruction and interface information of the first interface and from memory information stored in a terminal device. In the technical solution provided in the present application, a terminal device can automatically fill in information in a display interface on the basis of a user instruction, interface information of the display interface, and user memory information stored in the terminal device, and the present application provides a better usage experience for users.

Description

Device control method, electronic device, readable storage medium and chip

[0001] The present application claims priority to the Chinese Patent Application No. 202410809043.8, filed on June 20, 2024, entitled "Device Control Method, Electronic Device, Readable Storage Medium and Chip", and the Chinese Patent Application No. 202411091268.0, filed on August 08, 2024, entitled "Device Control Method, Electronic Device, Readable Storage Medium and Chip", the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a device control method, an electronic device, a readable storage medium and a chip. BACKGROUND

[0003] With the development of computer technology, voice assistants such as "Xiao Yi", "YOYO", "Siri" and the like gradually appear on terminal devices such as mobile phones and tablet computers. Users can interact with terminal devices through voice assistants. For example, users can control terminal devices to make calls, send messages, set alarms, play music, plan navigation, or control smart home devices through voice assistants. However, terminal devices currently only support processing simple user instructions such as making calls through voice assistants. If the user instruction given by the user is more complex, such as "help me fill out this form" or "help me order a milk tea", the terminal device will not be able to execute the user instruction, resulting in poor user experience. SUMMARY

[0004] The present application provides a device control method, an electronic device, a readable storage medium and a chip, which are used to solve the problem that terminal devices cannot execute complex user instructions in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a device control method, which is applied to a terminal device, and specifically includes: displaying a first interface, the first interface including a target area of to-be-filled information; in response to a first user instruction, automatically filling first target information in the target area; wherein the first target information is determined from memory information stored by the terminal device according to the first user instruction and interface information of the first interface.

[0006] By means of the device control method provided in the embodiment, the terminal device can perceive and understand the interface information of the display interface, and according to the interface information and the user instruction, obtain corresponding target information from the user memory information stored in the terminal device, and automatically fill the target information in the display interface, thus having good user experience.

[0007] In some embodiments, the first user instruction comprises: a voice control instruction; or a text control instruction; or a selection operation on the first control, the first control being displayed according to an index word input in the target area, the index word being associated with the first target information.

[0008] For example, the index word comprises: a person's name, an event name, a place name, a keyword, a commodity name, a date, etc. Taking the index word as a person's name as an example, the first target information is user information corresponding to the person's name.

[0009] In some embodiments, the first user instruction comprises a selection operation on the first control, and the automatic filling of the first target information in the target area in response to the first user instruction comprises: displaying the index word on the input method interface according to an input operation of the input method application; determining the index word as the first control; and automatically filling the first target information in the target area in response to the selection operation on the first control.

[0010] By means of the method provided in the embodiment, the user can quickly control the terminal device to automatically input target information in a display interface through an input method application. Taking the index word as a person's name "Zhang San" as an example, the user can input the person's name "Zhang San" in the input method application to automatically fill user information corresponding to "Zhang San" into the target area of the display interface, thus having good user experience.

[0011] In some embodiments, the first user instruction comprises a selection operation on the first control, and the automatic filling of the first target information in the target area in response to the first user instruction comprises: filling the index word in the target area of the first interface according to the input operation; displaying the first control on the first interface according to the index word; and automatically filling the first target information in the target area in response to the selection operation on the first control.

[0012] In the embodiment, the first control displayed according to the index word can be located at any position of the target area, for example, near the filled index word, and the embodiment does not limit this.

[0013] In some embodiments, in response to the first user instruction, automatically filling the first target information in the target area comprises: in response to the first user instruction, displaying candidate information on the first interface, the candidate information being determined from memory information stored in the terminal device according to the first user instruction and interface information of the first interface; determining the first target information from the candidate information according to a control operation; and automatically filling the first target information in the target area.

[0014] According to the method provided in the embodiments of the present application, the terminal device can select and correct the candidate information by the user before automatically filling the target information, that is, the terminal device automatically fills the information after the user confirms the information. It can be understood that the method can ensure the accuracy of information filling.

[0015] In some embodiments, the method further comprises: during the process of automatically filling the first target information in the target area, receiving a pause instruction; and in response to the pause instruction, pausing the terminal device to fill the first target information. That is, during the process of automatically filling the information by the terminal device, the user can control the terminal device to pause the filling at any time according to the needs, which has a good user experience.

[0016] In some embodiments, the method further comprises: after pausing the filling of the first target information or after filling the first target information, receiving a second user instruction; and in response to the second user instruction, automatically replacing the filled first target information with second target information; wherein the second target information is determined from memory information stored in the terminal device according to the second user instruction and interface information of the first interface. That is, the terminal device can automatically fill the information again according to the user instruction.

[0017] In some embodiments, before automatically filling the first target information in the target area in response to the first user instruction, the method further comprises: receiving an information storage instruction, the information storage instruction carrying an indication of the first target information; and storing the first target information. For example, the information storage instruction can be a voice control instruction or a text control instruction, and the terminal device can store the information carried in the information storage instruction as user memory information.

[0018] In some embodiments, before automatically filling the first target information in the target area in response to the first user instruction, the method further comprises: displaying a second interface, the second interface including the first target information; receiving an information storage instruction, the information storage instruction being used to instruct the terminal device to store the first target information; and storing the first target information in the memory information. That is, the terminal device can store the information displayed on the interface as user memory information according to the control operation of the user.

[0019] In a second aspect, the embodiments of the present application provide a device control method, applied to a terminal device, the method comprising: obtaining a first user instruction, the first user instruction being used to control the terminal device to perform a target operation; and in response to the first user instruction, automatically performing the target operation according to interface information of the terminal device and first target information, wherein the first target information is obtained from memory information stored by the terminal device according to the first user instruction.

[0020] By the device control method provided in the embodiments, after receiving the user instruction, the terminal device can determine and perform the target operation that meets the actual needs of the user according to the interface information of the display interface and the user memory information stored by the terminal device, thereby accurately executing the user instruction and having good user experience.

[0021] In some embodiments, the target operation comprises purchasing a target commodity, and the first target information comprises commodity information of the target commodity, such as a commodity name, a quantity, commodity attributes, an application program identifier, and a delivery address; and in response to the first user instruction, automatically performing the target operation according to the interface information of the terminal device and the first target information comprises: in response to the first user instruction, generating a purchase order of the target commodity according to the interface information of the terminal device and the commodity information of the target commodity.

[0022] In other words, the terminal device can help the user to automatically purchase a commodity, such as helping the user to order takeout, according to the user instruction and the interface information in combination with the user memory information stored by the terminal device, thereby having good user experience.

[0023] In some embodiments, the target operation comprises filling in information, and the first target information comprises content to be filled in.

[0024] In some embodiments, before obtaining the first user instruction, the method further comprises: displaying a first interface, the first interface comprising a target region of information to be filled in; and in response to the first user instruction, automatically performing the target operation according to the interface information of the terminal device and the first target information comprises: in response to the first user instruction, automatically filling in the first target information in the target region according to the interface information of the terminal device, wherein the first target information is determined from the memory information stored by the terminal device according to the first user instruction and the interface information of the first interface.

[0025] In the embodiments, the terminal device can perceive and understand the interface information of the display interface, and obtain target information that matches the interface information from the user memory information stored by the terminal device according to the user instruction and the interface information, and automatically fill in the target information in the display interface.

[0026] In some embodiments, in response to the first user instruction, the target operation is automatically performed according to the interface information of the terminal device and the first target information, including: in response to the first user instruction, displaying a first interface, the first interface including a target area of information to be filled; and automatically filling the first target information in the target area according to the interface information of the first interface, wherein the first target information is determined from memory information stored in the terminal device according to the first user instruction and the interface information of the first interface.

[0027] In some embodiments, the automatically filling the first target information in the target area includes: displaying candidate information on the first interface, wherein the candidate information is determined from memory information stored in the terminal device according to the first user instruction and the interface information of the first interface; determining the first target information from the candidate information according to a control operation; and automatically filling the first target information in the target area.

[0028] In some embodiments, the method further includes: after pausing the filling of the first target information, or after filling the first target information, receiving a second user instruction; in response to the second user instruction, automatically replacing the filled first target information with second target information; wherein the second target information is determined from memory information stored in the terminal device according to the second user instruction and the interface information of the first interface.

[0029] In some embodiments, the method further includes: during the process of automatically performing the target operation according to the first target information and the interface information of the terminal device, receiving a pause instruction; and in response to the pause instruction, pausing the execution of the target operation.

[0030] In some embodiments, before receiving the first user instruction, the method further includes: receiving an information storage instruction, the information storage instruction carrying the first target information; and storing the first target information.

[0031] In some embodiments, before receiving the first user instruction, the method further includes: displaying a second interface, the second interface including the first target information; receiving an information storage instruction, the information storage instruction being used to indicate storage of the first target information; and storing the first target information.

[0032] In a third aspect, the embodiments of the present application provide an electronic device, including: one or more processors and a memory; the memory is coupled to the one or more processors; the memory is configured to store computer program codes, the computer program codes including computer instructions, and the one or more processors are configured to invoke the computer instructions to enable the electronic device to perform the method shown in the first aspect or the second aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a chip, which is applied to an electronic device, and the chip comprises one or more processors, and the processor is configured to invoke a computer instruction to enable the electronic device to execute the method shown in the first aspect or the second aspect.

[0034] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method shown in the first aspect or the second aspect.

[0035] In a sixth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program, and when the computer program is run on an electronic device, the electronic device is enabled to implement the method shown in the first aspect or the second aspect.

[0036] It can be understood that the beneficial effects of the third aspect to the sixth aspect can be referred to the related description in the first aspect or the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0037] FIG. 1 is a schematic diagram of a voice assistant starting scene according to an embodiment of the present application;

[0038] FIG. 2 is a schematic diagram of a voice control scene according to an embodiment of the present application;

[0039] FIG. 3 is a schematic diagram of a device control system to which a device control method according to the present application is applied;

[0040] FIG. 4 is a flowchart of a device control method according to an embodiment of the present application;

[0041] FIG. 5 is a schematic diagram of a generation process of structured user memory information according to an embodiment of the present application;

[0042] FIGS. 6A-6D are schematic diagrams of storage processes of user memory information according to different embodiments of the present application;

[0043] FIG. 7 is a schematic flowchart of a device control method according to an embodiment of the present application;

[0044] FIGS. 8-11 are schematic diagrams of automatic filling processes of information according to different embodiments of the present application;

[0045] FIG. 12 is a schematic diagram of an interface including candidate information according to an embodiment of the present application;

[0046] FIG. 13 is a schematic diagram of a re-filling process of information according to an embodiment of the present application;

[0047] FIGS. 14-15 are schematic diagrams of automatic purchase processes of goods according to different embodiments of the present application;

[0048] Figure 16 is a schematic diagram of the structure of an electronic device provided in this embodiment;

[0049] Figure 17 is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0051] It should be understood that in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0052] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0053] With the development of computer technology, voice assistants such as "Xiaoyi", "YOYO" and "Siri" have gradually appeared on mobile phones, tablets and other terminal devices to assist users in human-computer interaction.

[0054] In this embodiment, the voice assistant, also known as an AI assistant, intelligent voice, digital assistant, virtual assistant, etc., can be understood as an information processing system that can recognize natural language input in the form of speech and / or text to infer the user's intent and perform corresponding actions based on the inferred user intent. Furthermore, the system can respond to the user's input through audible (e.g., speech) or visual (e.g., text, images) output formats.

[0055] To conserve power, voice assistants are typically in sleep mode, but users can activate them through specific control operations. Activating a voice assistant can also be called waking it up. For example, as shown in Figure 1, users can activate the voice assistant by saying wake-up words such as "Xiaoyi Xiaoyi" or "Hello YOYO." Alternatively, they can activate it by clicking the voice assistant icon, or by pressing and holding the power button. This embodiment does not limit the method of activating the voice assistant.

[0056] After activating the voice assistant, the terminal device can perform corresponding operations based on the user's commands. For example, the user can input the voice command "Play song A" to control the terminal device to play song A (see Figure 2). Or, the user can input the voice command "Call Lucy" to dial the contact Lucy's phone number. Alternatively, the user can input the voice command "Wake me up at 7:00 tomorrow morning" to set an alarm for 7:00 tomorrow morning. Or, the user can input the voice command "What holiday is today?" to provide a corresponding answer, such as "Today is New Year's Day." It can be seen that the voice assistant significantly improves the convenience of user interaction with the terminal device, bringing users a brand-new human-computer interaction experience.

[0057] In this embodiment, user instructions include simple user instructions and complex user instructions. Simple user instructions, also known as first-type user instructions, refer to user instructions that do not require understanding or perception of the display interface and can usually be completed with a single control operation, such as instructions to turn on Bluetooth, make a phone call, send a text message, play music, or set an alarm. Complex user instructions, also known as second-type user instructions, refer to user instructions that require understanding and perception of the display interface or require multiple control operations to be completed, such as "Help me fill out this form" and "Help me order a milk tea delivery."

[0058] Currently, terminal devices only support processing simple user commands through voice assistants. If the user gives a complex user command, such as "Help me fill out this form," the terminal device cannot dynamically generate a clear and executable processing plan based on the user command because the user command does not indicate the relevant information required to execute the command (such as the information required to fill out the form). In addition, the terminal device cannot understand and perceive the current interface information. Therefore, the terminal device cannot generate a control operation that meets the user's actual needs, that is, the terminal device cannot execute the user command correctly, resulting in a poor user experience.

[0059] Therefore, this application provides a device control method. In this method, for complex user commands, the terminal device can combine interface information and user memory information to control the device, thereby achieving control operations that meet the actual needs of the user and improving the user experience.

[0060] To facilitate understanding, the relevant terms and concepts involved in the embodiments of this application will be introduced below.

[0061] (1) Interface Information

[0062] Interface information includes content information and layout information of the displayed interface. Content information includes application identification (ID), window ID, application activation functions, and other application-related information. Layout information represents the hierarchical structure of the displayed interface, control information, atomic operation attributes of controls (such as whether a control is clickable or swipeable), icons, and text.

[0063] (2) Memorizing information

[0064] In this embodiment, the memory information is historical information obtained and stored by the user-authorized terminal device, including user memory information and interface fusion memory information.

[0065] User memory information refers to memory information related to user dialogue, user operations, user storage, or user habits. Examples include card information, important date information, and event information. Terminal devices can determine and store user memory information based on user instructions or by employing active memory technology, combined with a large-scale memory retrieval model.

[0066] Interface fusion memory information refers to memory information related to the historical display interface, including historical interface information (such as content information and layout information), user instructions and control operation sequences related to the historical display interface, etc.

[0067] (3) User instructions

[0068] In this embodiment, user instructions can also be referred to as user requests, query requests, etc., including voice control instructions, text control instructions, and control operations.

[0069] Voice control commands can be wake-up commands to activate a voice assistant or operation commands to control a terminal device to perform a target operation. For example, the voice command "Call Lucy for me".

[0070] Text control commands are typically used to control terminal devices to perform target operations and are entered by the user through a user dialog box of a voice assistant. For example, text control commands may be in the form of text such as "Help me fill out this form", "Order me a milk tea delivery", or "Set an alarm for 7 a.m. tomorrow".

[0071] Control operations can be user-initiated pressing of physical buttons, such as a long press of the power button. Alternatively, control operations can also be touch-based operations on controls within the display interface, such as clicking or swiping.

[0072] (4) Immediate user commands and non-immediate user commands

[0073] In this embodiment, user instructions are divided into immediate user instructions and non-immediate user instructions, depending on whether the terminal device needs to respond immediately.

[0074] Real-time user commands, also known as real-time queries, refer to commands that require an immediate output response from the terminal device, such as "Play me a song," "Set the living room air conditioner temperature to 26℃," "Call Lucy," "Fill out this form for me," or "Order me a milk tea delivery."

[0075] Non-real-time user commands, also known as non-real-time queries, refer to commands that require internal processing by the terminal device but do not require an output response at the moment, such as "Please save Lucy's phone number 135****2195".

[0076] Based on the above basic conceptual information, the technical solutions provided in the embodiments of this application will be described in detail below.

[0077] Figure 3 is a schematic diagram of the equipment control system to which the equipment control method provided in this application applies. The system includes a terminal device 100 and a server 200. The terminal device 100 and the server 200 can be wirelessly connected via wireless fidelity (Wi-Fi), cellular communication, or wired via network cable. This embodiment does not impose any restrictions on this connection.

[0078] Terminal device 100, also known as end-side device, includes, but is not limited to, mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other terminal devices. This application embodiment does not impose any restrictions on the specific type of terminal device.

[0079] In this embodiment, the terminal device includes an instruction acquisition module 110, an instruction parsing module 120, a user memory module 130, an interface fusion and understanding module 140, and an instruction execution module 150.

[0080] The instruction acquisition module 110 is used to acquire diverse user instructions, such as voice control instructions, text control instructions, and control operations shown in the above embodiments. After acquiring the user instructions, the instruction acquisition module 110 sends them to the instruction parsing module 120 for processing.

[0081] The instruction parsing module 120 is used to parse the meaning of a user instruction and determine its type after receiving it. Parsing the meaning of the instruction refers to determining the content of the user's instruction. For example, the instruction parsing module 120 can parse the user instruction using its own capabilities or with the assistance of the server 200; this embodiment does not impose any limitations on this. Determining the instruction type means determining whether the user instruction belongs to a first type or a second type of user instruction based on the content of the user's instruction.

[0082] In some embodiments, the instruction parsing module 120 may process different types of user instructions in different ways.

[0083] If the received user instruction is a type 1 user instruction, the instruction parsing module 120 determines the corresponding control operation based on the user instruction and sends the control operation to the instruction execution module 150 for processing. For example, the instruction parsing module 120 can determine the control operation corresponding to the user instruction using its own capabilities or with the help of the server 200. For instance, if the user instruction is "turn on Bluetooth," the instruction parsing module 120 generates a Bluetooth-turn-on instruction and sends it to the instruction execution module 150 for processing.

[0084] If the received user instruction is a type 2 user instruction, the instruction parsing module 120 can execute the corresponding operation based on the immediacy of the user instruction. The details are as follows.

[0085] When the user instruction is not immediate, such as "Please remember my daughter's birthday, September 1st," the instruction parsing module 120 generates structured user memory information based on the user instruction. For example, the instruction parsing module 120 can generate structured user memory information on its own. Alternatively, the instruction parsing module 120 can generate structured user memory information with the help of the server 200. For instance, the instruction parsing module 120 sends the user instruction or the parsed user instruction to the server 200, which then converts it into structured user memory information through the structured memory module 210 before sending it to the user memory module 130 of the terminal device 100 for storage (see the description of the structured memory module 210 below).

[0086] When the user instruction is an immediate user instruction but does not include memorized content, such as the user instruction "Help me fill out this form," the instruction parsing module 120 notifies the interface fusion and understanding module 140 to operate. The specific working process of the interface fusion and understanding module 140 is described below and will not be repeated here.

[0087] When the user instruction is an immediate user instruction and includes memory content, such as the user instruction "Please fill in Lucy's phone number 135****2195", the instruction parsing module 120 generates structured user memory information through its own / server 200 capabilities, and notifies the interface fusion and understanding module 140 to work.

[0088] User memory module 130 is used to store structured user memory information. For details on the structured user memory information, please refer to the relevant description in "Structured Memory Module 210" of "Server 200" below, which will not be limited here.

[0089] The interface integration and understanding module 140 includes a content understanding unit 141, a layout understanding unit 142, and an interface memory unit 143. Among them, the interface memory unit 143 is an optional unit.

[0090] The content understanding unit 141 is used to generate content information for the display interface based on the control information and application in the display interface. It should be noted that during the layout understanding process, if multiple windows exist in the display interface, the content understanding unit 141 can perform content understanding on each window separately.

[0091] The layout understanding unit 142 is used to identify objects in the current display interface, understand the relationships between these objects, and parse the overall meaning of the display interface, thereby mapping it into structured layout information. It should be noted that the layout understanding unit 142 can perform layout understanding entirely using its own capabilities, or it can perform layout understanding wholly or partially with the help of the server 200's capabilities; this embodiment does not impose any restrictions on this. During the process of performing layout understanding with the help of the server 200's capabilities, the layout understanding unit 142 sends the display interface to the server 200 and receives the layout information returned by the server.

[0092] In some embodiments, after the content understanding unit 141 and the layout understanding unit 142 obtain the content information and layout information of the display interface respectively, they send them together with the user instructions to the server 200 so that the server 200 can determine and issue the corresponding control instruction sequence.

[0093] In some embodiments, the interface fusion understanding module 140 further includes an interface memory unit 143. The interface memory unit 143 stores interface fusion memory information of historically displayed interfaces.

[0094] Based on this, after generating the content information of the current display interface, the content understanding unit 141 can send a retrieval instruction to the interface memory unit 143. This retrieval instruction carries the user instruction and the content information, and is used to instruct the interface memory unit 143 to search for interface fusion memory information that matches the current display interface. In addition, the layout understanding unit 142 also sends the layout information of the current display interface to the interface memory unit 143.

[0095] Accordingly, after receiving the retrieval command and layout information, the interface memory unit 143 merges the user command in the retrieval command, the content information of the currently displayed interface, and the layout information of the displayed interface to obtain the interface fusion understanding information of the current displayed interface. Next, based on this current interface fusion understanding information, it searches for similar interface fusion memory information in the interface memory unit 143. Then, it sends the current interface fusion understanding information and the interface fusion memory information together to the server 200, so that the server 200 can combine these two types of information to jointly determine and issue the corresponding control command sequence.

[0096] For example, the interface memory unit 143 can determine the similarity between the current interface fusion understanding information and each interface fusion memory information. The interface fusion memory information with the highest similarity greater than a threshold is then determined as the interface fusion memory information similar to the current interface fusion understanding information.

[0097] It should be noted that if the interface memory unit 143 does not find interface fusion memory information similar to the current interface fusion understanding information, it sends the interface fusion understanding information of the currently displayed interface to the server 200 for processing, so that the server 200 can directly generate a control instruction sequence based on the interface fusion understanding information.

[0098] The instruction execution module 150 is used to receive an instruction sequence and execute the control instructions in the instruction sequence in sequence to enable the terminal device to perform a specific function. For example, executing the control instructions in sequence can cause the terminal device to automatically fill in Lucy's passport information.

[0099] Server 200, also known as cloud server or cloud side, includes structured memory module 210, memory database 220, and memory reasoning framework module 230.

[0100] The structured memory module 210 is used to acquire, process, and store the user's memory information after obtaining the user's authorization. For example, the structured memory module 210 includes a memory acquisition unit 211, a memory understanding unit 212, and a memory storage unit 213.

[0101] The memory acquisition unit 211 is used to acquire user memory, that is, user usage information of the terminal device, or simply user usage information.

[0102] In some embodiments, user usage information includes conversation information between the user and the voice assistant. For example, "Xiaoyi, I like to drink milk tea from store A, but I don't like coffee," or "Please send Lucy a text message to wish her a happy birthday."

[0103] In other embodiments, user-accessed information includes information that the user actively submits to the voice assistant for storage. This information can be phone numbers, card information (such as ID card information, passport information), images, web pages, videos, text, documents, etc. For example, a user can submit information to the voice assistant for storage by using the voice control command "Save Lucy's ID card information in the display interface." Alternatively, a user can drag and drop the target information to be stored into the voice assistant's information storage entry point, thereby submitting the information to the voice assistant for storage.

[0104] In some other embodiments, user usage information includes factual content recorded based on the user's long-term (e.g., more than 7 days) usage habits of the voice assistant, such as phone numbers, card information (e.g., ID card information, passport information), event information, important date information (e.g., birthday, anniversary, meeting day), etc.

[0105] In other embodiments, user usage information includes information stored by the user on the terminal device, such as information stored by the user in a memo, photos, files, audio, video, web links, etc. stored by the user.

[0106] It should be noted that the memory acquisition unit 211 can acquire user information based on user instructions or through active memory acquisition. Active memory acquisition refers to the terminal device actively exploring the user's memory after user authorization, such as by using a memory retrieval model or active memory technology to acquire memo information or dialogue information between the user and the voice assistant, and then sending it to the memory understanding unit 212.

[0107] The memory comprehension unit 212 is used to understand user information and transform it into structured user memory information. This structured user memory information is used to abstract and summarize user information, including the user's memory categories, memory attributes (specific attributes based on the category), and memory content (see Table 1). It can be understood that the terminal device can infer the original memory content based on the structured information, facilitating subsequent querying, understanding, and display. The structured user memory information can be stored long-term in the terminal device, facilitating subsequent information retrieval and matching.

[0108] Table 1

[0109] Taking a passport as an example, Table 2 shows a structured user memory information about a passport.

[0110] Table 2

[0111] Referring to Examples 1 and 2 below, the memory understanding unit 212 can generate corresponding structured user memory information based on different user information (e.g., user instructions).

[0112] Example 1:

[0113] User instruction: Write this down, my daughter's birthday is September 1st every year.

[0114] Structured user memory information: [{"subCategory":"Birthday","importantDayType":"Daughter's Birthday","importantDate":"September 1st every year","subject":"Daughter"}]}.

[0115] Example 2:

[0116] User instruction: Please write down my daughter's ID number as 412728********3789.

[0117] Structured user memory information: [{"subCategory":"ID card","subject":"daughter","cardNumber":"412728********3789"}].

[0118] The memory storage unit 213 is used to encrypt and store structured user memory information for later use, with user authorization. For example, it can encrypt and store the structured user memory information described in Examples 1 and 2 above.

[0119] The memory database 220 is used to store the user's long-term memory information and short-term memory information. The long-term memory information includes encrypted user memory information (such as a mask or ID for the user memory information). The short-term memory information includes user operation records of the terminal device.

[0120] It should be noted that server 200 can synchronize users' long-term memory information across devices, so that electronic devices logged into the same user account (such as a Huawei account) can all use the user's long-term memory information.

[0121] The memory reasoning framework module 230 is used to generate a sequence of control instructions based on user commands and interface information.

[0122] In some embodiments, if the memory reasoning framework module 230 receives the interface fusion understanding information (including user instructions and interface information) of the current display interface but does not receive the historical layout fusion memory information, then the memory reasoning framework module 230 directly uses a reasoning model (such as a large language model (LLM)) to determine the corresponding control instruction sequence and sends the control instruction sequence to the instruction execution module 150 of the terminal device 100.

[0123] In other embodiments, the memory reasoning framework module 230 receives not only the interface fusion understanding information of the currently displayed interface, but also interface fusion memory information similar to the interface fusion understanding information. Based on this, the memory reasoning framework module 230 refers to the interface fusion memory information (such as the control instruction sequence therein), generates a control instruction sequence corresponding to the current interface fusion understanding information, and sends the control instruction sequence to the instruction execution module 150 of the terminal device 100.

[0124] It should be noted that the control instruction sequence in this embodiment includes multiple sequentially executed atomic control instructions. An atomic control instruction refers to one or a series of instructions that cannot be interrupted during execution. For example, these atomic control instructions include application programming interfaces (APIs), search operations, click operations, swipe operations, dialog operations, and confirm operations, etc. This embodiment does not limit the specific content of the control instructions.

[0125] In addition, the memory reasoning framework module 230 will generate interface fusion memory information based on the interface fusion understanding information and control command sequence of the currently displayed interface, and send it to the interface memory unit 143 of the terminal device 100 for storage for later use.

[0126] After introducing the equipment control system, the equipment control method provided in this embodiment will be described in detail below.

[0127] In this device control method, referring to S401-S402 in Figure 4, the terminal device first obtains a first user instruction, which instructs the terminal device to perform a target operation. For example, the target information could be filling in information, purchasing goods, etc. Subsequently, in response to the first user instruction, the terminal device automatically executes the target operation based on the interface information displayed on its screen and the first target information. The first target information is obtained by the terminal device from its stored user memory information based on the target operation.

[0128] The method provided in this embodiment enables the terminal device to combine interface information and user memory information after receiving the first user instruction, thereby realizing control operations that meet the user's actual needs and accurately executing the user instruction.

[0129] It is understood that in the device control method provided in this embodiment, the terminal device must first store user memory information before executing the aforementioned first user instruction based on the user memory information and interface information. The implementation process of these two parts will be explained in detail below in conjunction with user operations.

[0130] (a) Storing user memory information

[0131] In this embodiment, the terminal device can store user-remembered information according to the user's information storage instructions, or it can actively explore user-remembered information. This embodiment does not impose any limitations on this.

[0132] In scenarios where terminal devices store user-remembered information based on user information storage instructions, the information storage instructions include: voice control instructions, text control instructions input by the user through a voice assistant, or touch operations on the display interface.

[0133] Referring to Figure 5, after receiving the user's information storage instruction through the instruction acquisition module 110, the terminal device first sends it to the instruction parsing module 120 for user intent parsing to determine the information to be stored. Subsequently, the instruction parsing module 120 sends the information to be stored to the structured memory module 210 of the server 200 for processing. The structured memory module 210 processes the information into structured user memory information and returns it to the terminal device, which then stores it in the user memory module 130.

[0134] In some implementations, the information storage instruction carries the information to be stored. After receiving the information storage instruction, the terminal device converts the information carried in the instruction into structured user memory information and stores it locally.

[0135] For example, as shown in Figure 6A, a user issues a voice control command to a voice assistant (such as Xiaoyi): "Xiaoyi, Xiaoyi, please save my daughter's birthday, September 1st." Correspondingly, the terminal device receives this voice control command through the command acquisition module 110 and then sends it to the command parsing module 120 for user intent parsing. The command parsing module 120 sends the information to be stored, "my daughter's birthday, September 1st," to the structured memory module 210 of the server 200 for processing. The structured memory module 210 processes this information into structured user memory information and returns it to the terminal device, which then stores it in the user memory module 130.

[0136] For example, as shown in Figure 6B, a user sends a text control command to a voice assistant (such as Xiaoyi) through a dialog box: "Help me save my daughter's birthday, September 1st." The terminal device converts "my daughter's birthday, September 1st" into structured user memory information and stores it in the terminal device.

[0137] For example, as shown in Figure 6C, the electronic device's display interface shows information to be stored, such as Zhang San's information. Based on this, in response to the user's drag-and-drop operation of the information to be stored, the terminal device displays a storage control, such as an "AI Storage" control. After detecting that the user has dragged the information to be stored to the storage control and released it, the terminal device processes Zhang San's information into structured user memory information and stores it in the terminal device.

[0138] In other implementations, the information storage instruction does not carry the information to be stored, but it instructs the terminal device to store the information to be stored in the currently displayed interface (e.g., the second interface). Based on this, the terminal device stores the target information in the second interface as user memory information according to the information storage instruction.

[0139] For example, as shown in Figure 6D, the electronic device's display interface shows information to be stored, such as Zhang San's ID card information. Based on this, the user issues a voice control command to the voice assistant (such as Xiaoyi): "Xiaoyi, Xiaoyi, help me store Zhang San's information." The terminal device receives this voice control command through the command acquisition module 110, and then sends it to the command parsing module 120 for user intent parsing. The command parsing module 120 processes the information of Zhang San to be stored into structured memory information through the structured memory module 210 of the server 200, and stores it in the user memory module 130.

[0140] In scenarios where a terminal device obtains user authorization and actively explores the user's memory information, it can employ techniques such as memory retrieval models and active memory technology to convert information stored on the terminal device or factual content related to the user's usage habits into user memory information for storage. It's understandable that during this process of the terminal device actively exploring user memory information, user intervention is typically not required.

[0141] (ii) Execute the first user instruction based on user memory information and interface information.

[0142] Figure 7 is a schematic flowchart of a device control method provided in an embodiment of this application. Referring to Figure 7, the method involves a terminal device executing complex user instructions, specifically including the following steps S701 to S707.

[0143] S701, the terminal device obtains a first user instruction, which is used to instruct the terminal device to perform a target operation.

[0144] The terminal device can obtain the first user instruction through the instruction acquisition module 110. In this embodiment, the terminal device supports obtaining diverse user instructions. For example, the first user instruction may be a voice control instruction or a text control instruction obtained through a voice assistant, or it may be a user's selection operation of application controls on the terminal device, or a pressing operation of physical buttons on the terminal device, etc. This embodiment does not impose specific restrictions on its type.

[0145] Since the first user instruction is issued by the user based on their actual control needs, and given the diversity of user needs, the target operation that the first user instruction instructs the terminal device to perform also varies. For example, the target operation might be controlling the terminal device to fill in information (such as passport information, ID card information, etc.) on the display interface; or, the target operation might be controlling the terminal device to purchase goods (such as daily necessities, food, takeout, books, etc.) through an application, where purchasing goods can be understood as generating a purchase order for the goods.

[0146] S702, the terminal device determines whether the first user instruction is a second type of user instruction.

[0147] After receiving the first user instruction through the instruction acquisition module 110, the terminal device sends it to the instruction parsing module 120 for instruction parsing to determine the instruction type of the first user instruction. If the first user instruction belongs to the first type of user instruction (i.e., a simple user instruction), the terminal device directly generates and executes the corresponding control operation based on the result of the first user instruction to realize the user's intention. If the first user instruction is a second type of user instruction (i.e., a complex user instruction), the terminal device executes subsequent steps S703 to S707.

[0148] S703, if the first user instruction is a second type of user instruction, the terminal device determines the interface information of the currently displayed interface.

[0149] In this embodiment, the first user instruction may not be explicit, such as "Help me fill out a form" or "Help me order milk tea delivery". Therefore, the terminal device needs to understand the current display interface and then combine the user instruction with the understood interface information to determine the specific content to be filled in.

[0150] In S703, for example, after determining that the first user instruction is a second type of user instruction, the terminal device determines its interface information through the interface fusion understanding module 140. The interface information includes the content information and layout information of the displayed interface.

[0151] In some embodiments, the interface fusion understanding module 140 may also search its interface memory unit 143 for interface fusion memory information similar to the first user instruction and interface information.

[0152] S704, the terminal device sends the first user command and interface information to the server.

[0153] In some embodiments, if the interface fusion understanding module 140 of the terminal device also finds interface fusion memory information similar to the first user instruction and interface information, then the terminal device sends the interface fusion memory information together with the first user instruction and interface information to the server.

[0154] S705: The server generates a sequence of control commands based on the first user instruction and interface information.

[0155] The server's memory reasoning framework module 230 can generate a sequence of control commands based on the received information. The specific process of generating the control command sequence can be found in the functional introduction section of the memory reasoning framework module 230 above, and will not be repeated here.

[0156] Taking the scenario of filling in passenger information as an example, the control instruction sequence generated by the server's memory reasoning framework module 230 includes multiple sequentially executed atomic control instructions. Referring to Table 3, this control instruction sequence includes control instruction 1, control instruction 2, and control instruction 3. Specifically, control instruction 1 controls the terminal device to fill in the passenger's name on the display interface, control instruction 2 controls the terminal device to fill in the passenger's ID number on the display interface, and control instruction 3 controls the terminal device to fill in the passenger's phone number on the display interface.

[0157] Table 3 Control Command Sequence

[0158] S706, the server sends a sequence of control commands to the terminal device.

[0159] S707, the terminal device obtains the first target information from the user memory information according to the control instruction sequence, and executes the control instruction sequence according to the first target information to achieve the target operation.

[0160] It should be noted that since the server does not have user memory information, the control instruction sequence it returns does not include the first target information required for instruction execution. The terminal device needs to obtain the first target information from its stored user memory information before it can complete the corresponding control function according to the control instruction sequence.

[0161] In some embodiments, the target operation indicated by the first user instruction is to fill in information on the display interface. Therefore, the terminal device obtains the first target information as the information to be filled in, such as name, ID number, or phone number.

[0162] In other embodiments, the target operation indicated by the first user instruction is: to purchase a target product. Then, the terminal device obtains the product information of the target product as the first target information, such as the product name, quantity, product attributes, application identifier, and shipping address.

[0163] Taking the control execution sequence shown in Table 3 as an example, the instruction execution module 150 retrieves the corresponding user information from the user memory module 130 according to the control instruction sequence and fills it into the corresponding input position on the display interface. For example, according to user instruction sequence 2, it retrieves Zhang San's ID card number "610******8222" from the user memory module 130 and fills it into the ID card number input slot. Or, according to user instruction sequence 3, it retrieves Zhang San's phone number "135***2195" from the user memory module 130 and fills it into the phone number input slot.

[0164] In addition, after successfully executing the target operation, the terminal device can store the interface information of the currently displayed interface, the first user instruction, and the control instruction sequence as interface fusion memory information in the interface memory unit 143 for reference when the terminal device processes similar user instructions in the future, so as to improve the accuracy of the control instruction sequence.

[0165] Through the device control method provided in this embodiment, after receiving complex user instructions, the terminal device can understand the current interface information by combining interface analysis (including content analysis and layout analysis) technology, and dynamically generate a clear and executable control plan by combining user memory information, so as to realize continuous control operation that meets the actual needs of the user and accurately execute user instructions.

[0166] As described above, the first user instruction can control the terminal device to perform various target operations, such as controlling the terminal device to automatically fill in information on the display interface or controlling the terminal device to automatically purchase target goods in the application. The device control method provided in this application embodiment will be specifically described below with reference to these two examples.

[0167] (1) Automatically fill in information on the display interface.

[0168] This embodiment supports diverse user commands, such as voice control commands, text control commands, or touch control operations. The following example, using the scenario of a user adding passenger Zhang San's information on the "Add Passenger" interface of a ticketing application, illustrates the process of the terminal device automatically filling in information on the display interface.

[0169] In some embodiments, the terminal device may first display an interface for filling in information, such as a first interface, which includes a target area for the information to be filled in. Subsequently, in response to a first user instruction, the terminal device automatically fills in the first target information within the target area based on the interface information of the first interface. The first target information is determined from memory information stored in the terminal device based on the first user instruction and the interface information of the first interface.

[0170] Example 1: Automatically fill in information on the display interface according to voice control commands.

[0171] Referring to Figures 8(a) to (c), firstly, the terminal device displays the "Add Passenger" interface based on the user's operation. The information to be filled in on this interface includes "Name," "ID Number," and "Phone Number." Subsequently, in response to the user's voice control command "Xiaoyi, Xiaoyi, help me fill in Zhang San's information" input through the voice assistant "Xiaoyi," the terminal device automatically fills in the name "Zhang San," the ID number "610******8222," and the phone number "135***2195."

[0172] Optionally, as shown in Figure 8(b), the terminal device may display a prompt message, such as "Filling out the form for you...", during the automatic information filling process, to indicate to the user that the terminal device is processing the information.

[0173] Example 2: Automatically fill in information on the display interface based on text control instructions.

[0174] Referring to Figures 9(a) to (c), firstly, the terminal device displays the "Add Passenger" interface based on the user's operation. The information to be filled in on this interface includes "Name," "ID Number," and "Phone Number." Subsequently, in response to the user's text control command "Help me fill in Zhang San's information" entered in the voice assistant "Xiao Yi's" dialog box, the terminal device automatically fills in the name "Zhang San," ID number "610******8222," and phone number "135***2195" on the "Add Passenger" interface.

[0175] Example 3: Based on the selection operation of the first control, information is automatically filled in on the display interface.

[0176] In this embodiment, the terminal device first fills in index terms on the first interface based on the user's input. For example, index terms include: person names, event names, location names, keywords, product names, dates, etc. After detecting the index terms, the terminal device displays a first control on the first interface. This first control indicates that the terminal device stores the first target information corresponding to the index term. In response to a selection operation on the first control, the first target information is automatically filled in within the target area.

[0177] Referring to Figures 10(a) to (d), firstly, the terminal device displays the "Add Passenger" interface based on the user's operation. The information to be filled in on this interface includes "Name," "ID Number," and "Phone Number." Then, based on the user's input, the terminal device fills in the index term "Zhang San" in the name field using the input method. After detecting the index term "Zhang San," the terminal device displays a first control on the interface to retrieve relevant information about Zhang San, such as an "AutoFill" control. In response to the user's selection of the "AutoFill" control, the terminal device automatically fills in the name "Zhang San," ID number "610******8222," and phone number "135***2195" on the "Add Passenger" interface.

[0178] Referring to Figures 11(a) to (c), firstly, the terminal device displays the "Add Passenger" interface based on the user's operation. The information to be filled in on this interface includes "Name," "ID Number," and "Phone Number." Then, based on the user's input, the terminal device searches for the index term "Zhang San" in the input method interface and selects this search term as the first control. In response to the user's selection of the first control "Zhang San," the terminal device automatically fills in the name "Zhang San," ID number "610******8222," and phone number "135***2195" on the "Add Passenger" interface.

[0179] In some other embodiments, the terminal device first receives a first user instruction, and then displays a first interface according to the first user instruction. This first interface includes a target area for information to be filled in. Finally, the terminal device automatically fills in the first target information within the target area based on the interface information of the first interface. For example, the first user instruction could be "Help me fill in Form A on my desktop".

[0180] As can be seen, through the device control method provided in this embodiment, the terminal device can achieve "one-click automatic form filling" by sensing and understanding the display interface and combining it with the user's memory information, which has a better user experience.

[0181] The following details other possible control operations involved in the "one-click auto-fill" process.

[0182] After receiving the first user instruction to fill in information, if the terminal device analyzes and finds that the first user instruction is unclear or the user's memory information is missing, causing the terminal device to be unable to generate a control instruction sequence, then the terminal device can further question the user to obtain relevant information.

[0183] In some embodiments, the terminal device receives a first user instruction to fill in information, such as "Please fill in Zhang San's information for me," but cannot determine the area where the information needs to be filled in. For example, the terminal device is currently displaying a desktop interface, and there is a table A on the desktop, but table A is not open; that is, the current display interface of the terminal device does not include the target area for the information to be filled in. Based on this, the terminal device can further inquire about the location where the information needs to be filled in, for example, the terminal device can ask the user, "Where do I fill this in?" Assuming the user's answer is "Fill it in table A on the desktop," then the terminal device opens table A on the desktop and fills in Zhang San's information in table A.

[0184] In other embodiments, after receiving a first user instruction to fill in information, if the terminal device discovers that the user's memory information is missing information to be filled in, the terminal device can ask the user for the missing information. For example, in a scenario where the user instructs the terminal device to fill in Zhang San's information, and it is detected that all or part of Zhang San's information is missing from the user's memory information, the terminal device can ask the user through a voice assistant, "What are Zhang San's ID number and phone number?" After receiving the missing information from the user, the terminal device can fill in Zhang San's information on the current display interface.

[0185] To ensure the accuracy of user information entry, after receiving the control command list and identifying the user information to be entered, the terminal device can display this information as candidate information on the interface for further confirmation or modification by the user. Subsequently, in response to the user's selection of the first target information from the candidate information, the terminal device enters the first target information into the corresponding position on the display interface.

[0186] Referring to Figure 12, taking the example of a user adding passenger Zhang San's information on the "Add Passenger" interface of a ticketing application, the terminal device can first display a prompt box. This prompt box includes candidate information, a selection control for each candidate information, a cancel control, and a confirmation control. For example, the candidate information includes the name "Zhang San," "Mobile Number 135****2195," and "ID Number 610******8222." The selection control is used to determine whether to fill in the corresponding candidate information. The cancel control is used to cancel filling in user information on the current interface. The confirmation control can be the "Start Filling" control shown in the figure, used to control the terminal device to fill in the candidate information selected by the user. In response to the user's selection operation on the confirmation control, the terminal device confirms the candidate information selected by the user as the first target information and fills it in the corresponding slot on the current display interface.

[0187] Furthermore, during the process of filling in information using the terminal device, users may need to pause or re-enter their information due to reasons such as incorrect instructions or changes in the filling plan. In this embodiment, the terminal device can also respond to the user's request to pause or re-enter their information, as detailed below.

[0188] In some embodiments, if the terminal device receives a pause instruction during the automatic filling of the first target information, such as "Hey Hey Hey, pause filling" or "Hey Hey Hey, don't fill it in yet," then in response to the pause instruction, the terminal device pauses filling in the first target information. For example, if the terminal device receives a pause instruction while filling in Zhang San's information on the "Add Passenger" interface, it will pause filling in Zhang San's information.

[0189] In other embodiments, the terminal device receives a second user instruction after pausing the entry of the first target information or after completing the entry of the first target information. In response to the second user instruction, the terminal device automatically replaces the entered first target information with the second target information. The second target information is determined from user memory information stored in the terminal device based on the second user instruction and the interface information of the first interface.

[0190] For example, referring to Figures 13(a) to (c), after the terminal device fills in Zhang San's information according to the first user instruction, it receives a second user instruction, "Help me re-fill in Li Si's information." Then, the terminal device deletes Zhang San's information from the display interface and retrieves Li Si's information from the user memory information stored in the terminal device according to the received control instruction sequence, automatically filling it into the corresponding information slot on the current display interface. For example, Li Si's information includes: name "Li Si," ID number "610******0335," and phone number "150****6930."

[0191] Optionally, the terminal device can also update some of the first target information that has been filled in according to user instructions, such as updating Zhang San's mobile phone number from "135****2195" to "135****6789".

[0192] It should be noted that the information processing of the terminal device in response to the second user instruction to fill in the second target information is the same as the processing process in response to the first user instruction to fill in the second target information. For details, please refer to steps S701 to S707 shown above, which will not be repeated here.

[0193] (2) Automatic purchase of goods

[0194] The following section uses different forms of user instructions, taking the example of a user instructing a terminal device to "order milk tea delivery," to illustrate the process of a terminal device automatically purchasing goods based on user instructions.

[0195] Example 1: Purchase goods according to voice control instructions.

[0196] In response to a user's voice control command, "Order me a milk tea delivery," input through the voice assistant "Xiaoyi," the voice assistant replies that it will place the order based on the user's usual habits. For example, it might reply, "Okay, I will place the order for you based on your usual habits. You usually prefer to use app A to place orders, so we will use app A to place the order for you." Subsequently, the terminal device automatically performs operations such as opening app A, entering the online store, selecting the desired milk tea, and selecting milk tea attributes based on the user's usual habits, finally generating a milk tea purchase order.

[0197] Example 2: Purchase goods based on text control instructions.

[0198] As shown in Figure 14, in response to the user's text control command "Order me a milk tea delivery" entered through the voice assistant "Xiaoyi" in the dialog box, the voice assistant replies in the dialog box that it will place the order according to the user's usual habits. For example, it replies, "Okay, I will place the order for you according to your usual habits. You usually like to use app A to place orders, so we will use app A to place the order for you." Subsequently, the terminal device opens the food delivery app, enters the online store, selects the target milk tea, and finally generates a milk tea purchase order based on the user's usual habits.

[0199] After a purchase order is generated on the terminal device, if the user has authorized the application to make a password-free payment, the application will automatically pay for the order. If the user has not authorized the application to make a password-free payment, the user will need to manually control the payment of the order.

[0200] It should be noted that during the purchase process, terminal devices typically display multiple levels of interfaces. The terminal device needs to perform a fusion understanding of each level of the interface before determining the corresponding control command sequence. Therefore, after each new interface is displayed, the terminal device needs to use the interface fusion understanding module 140 to determine the interface information and send it to the server's memory reasoning framework module 230. This allows the memory reasoning framework module 230 to determine the control command sequence for the newly displayed interface based on the interface information and the user's instructions.

[0201] It is understandable that by repeatedly executing the above process of "interface understanding - executing the current interface control instruction sequence - displaying the new interface" multiple times, the terminal device can successfully generate a product purchase order.

[0202] For example, as shown in Figure 15, when the terminal device receives the user's text control command "Order me a milk tea delivery" while displaying the desktop, the terminal device first generates a first control command sequence based on the text control command and the desktop interface information, including the control command to open the food delivery application A. After executing the control command to open the food delivery application A, the terminal device displays the store selection interface of food delivery application A, which includes multiple stores such as store 1, store 2, and store 3. Subsequently, the terminal device generates a second control command sequence based on the interface information of the store selection interface, including the control command to open store 1. After executing the control command to open store 1, the terminal device displays the product selection interface of store 1, which includes multiple products such as product A, product B, and product C. Then, based on the interface information of the product selection interface, the terminal device generates a third control command sequence, including the control command to select product B. After executing the control command to select product B, the terminal device displays the attribute selection interface of product B, which, for example, includes milk tea cup type, temperature, and sweetness. Next, based on the interface information of the attribute selection interface, the terminal device generates a fourth control instruction sequence, including: cup type selection instruction, temperature selection instruction, and sweetness selection instruction. The terminal device executes each of these selection instructions sequentially to determine the product attributes. Finally, based on the product attributes, the terminal device generates and executes a product purchase instruction, creating a product purchase order.

[0203] As can be seen, during the above process, the terminal device can perform personalized control operations based on user preferences, habits, or actions, resulting in a better user experience.

[0204] To further enhance the intelligent execution of user commands, the terminal device can optimize the generated atomic control command sequence based on user feedback and full-scene control perception results. User feedback includes, but is not limited to, user ratings, evaluations, likes / dislikes, etc., while full-scene control perception results include the fusion perception results of control changes and interface information. For example, if a control action fails or does not proceed as expected, the control result is stored in the interface memory unit 143. During the next control execution, a machine learning algorithm can be used to learn from this error and avoid similar problems in the future.

[0205] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0206] This application also provides an electronic device, which includes one or more processors and a memory; the memory is coupled to one or more processors; the memory is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the device control methods shown in the above embodiments.

[0207] Figure 16 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may include a processor 1610, an external memory interface 1620, an internal memory 1621, a universal serial bus (USB) interface 1630, a charging management module 1640, a power management module 1641, a battery 1642, antenna 1, antenna 2, a mobile communication module 1650, a wireless communication module 1660, an audio module 1670, a speaker 1670A, a receiver 1670B, a microphone 1670C, a headphone jack 1670D, a sensor module 1680, buttons 1690, a motor 1691, an indicator 1692, a camera 1693, a display screen 1694, and a subscriber identification module (SIM) card interface 1695, etc.

[0208] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, when the electronic device is a mobile phone or a tablet computer, it may include all the components illustrated, or it may include only some of the components illustrated.

[0209] Processor 1610 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0210] The charging management module 1640 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0211] The power management module 1641 is used to connect the battery 1642, the charging management module 1640, and the processor 1610. The power management module 1641 receives input from the battery 1642 and / or the charging management module 1640, as well as power to the processor 1610, internal memory 1621, external memory, display 1694, camera 1693, and wireless communication module 1660.

[0212] The wireless communication function of electronic devices can be implemented through antenna 1, antenna 2, mobile communication module 1650, wireless communication module 1660, modem processor, and baseband processor.

[0213] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0214] The mobile communication module 1650 can provide solutions for wireless communication applications in electronic devices, including 2G / 3G / 4G / 5G. The mobile communication module 1650 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 1650 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 1650 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0215] The wireless communication module 1660 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 1660 can be one or more devices integrating at least one communication processing module. The wireless communication module 1660 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 1610. The wireless communication module 1660 can also receive signals to be transmitted from processor 1610, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0216] Electronic devices implement display functions through a GPU, a display screen 1694, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 1694 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 1610 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0217] The display screen 1694 is used to display images, videos, etc. The display screen 1694 includes a display panel. In some embodiments, the electronic device may include one or N display screens 1694, where N is a positive integer greater than 1.

[0218] Electronic devices can achieve shooting functions through ISP, camera 1693, video codec, GPU, display 1694 and application processor.

[0219] The ISP (Image Signal Processor) is used to process data fed back from the camera 1693. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The camera 1693 is used to capture still images or videos.

[0220] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0221] Video codecs are used to compress or decompress digital video.

[0222] The external storage interface 1620 can be used to connect external storage cards, such as Micro SD cards, to expand the storage capacity of electronic devices. The external storage card communicates with the processor 1610 through the external storage interface 1620 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0223] Internal memory 1621 can be used to store executable program code, which includes instructions. Processor 1610 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 1621. Internal memory 1621 may include a program storage area and a data storage area.

[0224] In addition, the internal memory 1621 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0225] Electronic devices can implement audio functions through audio modules 1670, speakers 1670A, receivers 1670B, microphones 1670C, headphone jacks 1670D, and application processors.

[0226] The audio module 1670 is used to convert digital audio signals into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 1670 can also be used for encoding and decoding audio signals.

[0227] The speaker 1670A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic devices can listen to music or make hands-free calls through the speaker 1670A; for example, the speaker can play the comparison analysis results provided in the embodiments of this application.

[0228] The receiver 1670B, also known as a "handpiece," is used to convert audio electrical signals into sound signals. When an electronic device answers a phone call or voice message, the receiver 1670B can be brought close to the ear to hear the voice.

[0229] The microphone 1670C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to the microphone 1670C, inputting the sound signal into the microphone 1670C.

[0230] The 1670D headphone jack is used to connect wired headphones.

[0231] The sensor module 1680 may include a pressure sensor 1680A, a gyroscope sensor 1680B, a barometric pressure sensor 1680C, a magnetic sensor 1680D, an accelerometer sensor 1680E, a proximity sensor 1680F, a proximity light sensor 1680G, a fingerprint sensor 1680H, a temperature sensor 1680J, a touch sensor 1680K, an ambient light sensor 1680L, a bone conduction sensor 1680M, etc.

[0232] Buttons 1690 include power buttons, volume buttons, etc. Buttons 1690 can be mechanical buttons or touch buttons. Electronic devices can receive button inputs and generate key signal inputs related to user settings and function control.

[0233] Motor 1691 can generate vibration alerts. Motor 1691 can be used for incoming call vibration alerts or for touch vibration feedback.

[0234] Indicator 1692 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0235] The SIM card interface 1695 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 1695 to achieve contact and separation with the electronic device.

[0236] This application also provides a chip, as shown in FIG17, which includes a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the device control methods in the above embodiments.

[0237] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the device control methods provided in the above embodiments.

[0238] This application also provides a computer program product, which includes a computer program that, when run by an electronic device, enables the electronic device to implement the device control methods provided in the above embodiments.

[0239] It should be understood that the processor mentioned in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0240] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0241] In the embodiments provided in this application, the division of each framework or module is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple frameworks or modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0242] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0243] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0244] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0245] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A device control method, characterized in that, Applied to a terminal device, the method includes: Display a first interface, which includes a target area for information to be filled in; In response to a first user instruction, first target information is automatically filled in within the target area; wherein the first target information is memory information associated with the first user instruction and the interface information of the first interface.

2. The method according to claim 1, characterized in that, The first user instruction includes: Voice control commands; or, Text control commands; or, The selection operation of the first control is performed, wherein the first control is displayed based on the index term entered in the target area, and the index term is associated with the first target information.

3. The method according to claim 1 or 2, characterized in that, The first user instruction includes a selection operation on the first control, and the step of automatically filling in the first target information in the target area in response to the first user instruction includes: Based on the input operation in the input method application, display index words on the input method interface; The index term is determined as the first control; In response to the selection operation of the first control, the first target information is automatically filled in the target area.

4. The method according to claim 1 or 2, characterized in that, The first user instruction includes a selection operation on the first control, and the step of automatically filling in the first target information in the target area in response to the first user instruction includes: Based on the input operation, fill in the index words in the target area of ​​the first interface; Display a first control on the first interface according to the index term; In response to the selection operation of the first control, the first target information is automatically filled in the target area.

5. The method according to any one of claims 1 to 4, characterized in that, In response to a first user instruction, first target information is automatically filled in within the target area, including: In response to a first user instruction, candidate information is displayed on the first interface, the candidate information being determined from memory information stored in the terminal device based on the first user instruction and the interface information of the first interface; Based on the control operation, the first target information is determined from the candidate information; The first target information is automatically filled in within the target area.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: During the process of automatically filling in the first target information within the target area, a pause command is received. In response to the pause command, the terminal device pauses filling in the first target information.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: After pausing the filling of the first target information, or after the first target information has been filled in, a second user instruction is received; In response to the second user instruction, the first target information that has been filled in is automatically replaced with the second target information; The second target information is determined from the memory information stored in the terminal device based on the second user instruction and the interface information of the first interface.

8. The method according to any one of claims 1 to 7, characterized in that, Before automatically filling in the first target information within the target area in response to the first user instruction, the method further includes: Receive an information storage instruction, wherein the information storage instruction carries the first target information; Store the first target information.

9. The method according to any one of claims 1 to 7, characterized in that, Before automatically filling in the first target information within the target area in response to the first user instruction, the method further includes: A second interface is displayed, which includes the first target information; Receive an information storage instruction, the information storage instruction being used to instruct the terminal device to store the first target information; The first target information is stored in the memory information.

10. A device control method, characterized in that, Applied to a terminal device, the method includes: Obtain a first user instruction, which is used to control the terminal device to perform a target operation; In response to the first user instruction, the target operation is automatically executed based on the interface information of the terminal device and the first target information, wherein the first target information is memory information associated with the first user instruction and the interface information.

11. The method according to claim 10, characterized in that, The target operation includes purchasing a target product, and the first target information includes product information of the target product; the step of automatically executing the target operation in response to the first user instruction, based on the interface information of the terminal device and the first target information, includes: In response to the first user instruction, a purchase order for the target product is generated based on the interface information of the terminal device and the product information of the target product.

12. The method according to claim 11, characterized in that, The product information includes: product name, quantity, product attributes, application identifier, and shipping address.

13. The method according to claim 10, characterized in that, The target operation includes filling in information, and the first target information includes the content to be filled in.

14. The method according to claim 13, characterized in that, Prior to obtaining the first user instruction, the method further includes: displaying a first interface, the first interface including a target area for information to be filled in; and, The step of automatically executing the target operation in response to the first user instruction, based on the interface information of the terminal device and the first target information, includes: automatically filling in the first target information in the target area in response to the first user instruction, based on the interface information of the terminal device, wherein the first target information is determined from the memory information stored in the terminal device based on the first user instruction and the interface information of the first interface.

15. The method according to claim 13, characterized in that, The step of automatically executing the target operation in response to the first user instruction, based on the interface information of the terminal device and the first target information, includes: In response to the first user instruction, a first interface is displayed, the first interface including a target area for information to be filled in; Based on the interface information of the first interface, the first target information is automatically filled in the target area, wherein the first target information is determined from the memory information stored in the terminal device based on the first user instruction and the interface information of the first interface.

16. The method according to claim 14 or 15, characterized in that, The automatic filling of first target information within the target area includes: Candidate information is displayed on the first interface, wherein the candidate information is determined from memory information stored in the terminal device based on the first user instruction and the interface information of the first interface. Based on the control operation, the first target information is determined from the candidate information; The first target information is automatically filled in within the target area.

17. The method according to any one of claims 13 to 16, characterized in that, The method further includes: After pausing the filling of the first target information, or after the first target information has been filled in, a second user instruction is received; In response to the second user instruction, the first target information that has been filled in is automatically replaced with the second target information; The second target information is determined from the memory information stored in the terminal device based on the second user instruction and the interface information of the first interface.

18. The method according to any one of claims 10 to 17, characterized in that, The method further includes: During the process of automatically executing the target operation based on the first target information and the interface information of the terminal device, a pause command is received; In response to the pause command, the execution of the target operation is paused.

19. The method according to any one of claims 10 to 18, characterized in that, Before receiving the first user instruction, the method further includes: Receive an information storage instruction, wherein the information storage instruction carries the first target information; Store the first target information.

20. The method according to any one of claims 10 to 19, characterized in that, Before receiving the first user instruction, the method further includes: A second interface is displayed, which includes the first target information; Receive an information storage instruction, the information storage instruction being used to instruct the storage of the first target information; Store the first target information.

21. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors; The memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9 or 10 to 20.

22. A chip, characterized in that, The chip is used in an electronic device, the chip including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9 or any one of claims 10 to 20.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 9 or any one of claims 10 to 20.

Images