Information processing methods, systems, electronic devices, storage media and program products

Abstract

This application relates to the field of electronic devices, and discloses an information processing method, system, electronic device, storage medium, and program product. In the method provided by this application, a first user instruction is detected. If the first instruction lacks first parameter information, a first query message is output. The first query message is used to query the user for the first parameter information, which includes at least one of application information and execution operation information (second parameter information). A second user instruction is detected, which is used to instruct the user to use the first parameter information. A first tool can be determined based on the first instruction and the second instruction, and a first task can be executed based on the first tool. Thus, by actively probing, the success rate of user instruction execution can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to an information processing method, system, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of e-commerce, shopping functions have become more diversified. Users typically need to perform multiple operations to complete tasks such as placing an order, leaving a review, and searching. To simplify user operations, voice assistants (or shopping assistants) exist in electronic devices, which can automatically execute some operations during the shopping process based on user commands, such as shopping, leaving a review, or searching. However, due to the diversity and complexity of user commands, the success rate of voice assistants in executing user commands is currently relatively low. Summary of the Invention

[0003] This application provides an information processing method, system, electronic device, storage medium, and program product to improve the success rate of voice assistants executing user commands in shopping scenarios.

[0004] In a first aspect, embodiments of this application provide an information processing method for a first electronic device. The method includes: detecting a first instruction from a user and sending the first instruction to a second electronic device; receiving first query information from the second electronic device and outputting the first query information, wherein the first query information is sent by the second electronic device to the first electronic device when it is determined that the first instruction lacks first parameter information, and the first query information is used to query the user for first parameter information, wherein the first parameter information includes at least one of application information and execution operation information, and second parameter information; detecting a second instruction from a user and sending a second instruction to the second electronic device, wherein the second instruction is used to indicate the first parameter information; receiving a first tool from the second electronic device and performing a first task based on the first tool, wherein the first tool is determined by the second electronic device based on the first instruction and the second instruction.

[0005] Based on the above scheme, the first electronic device can proactively ask follow-up questions to obtain the missing information when the user's instruction lacks application information and parameter information (such as the purchased goods) in the execution operation information. Based on the user's reply and the original instruction, the corresponding task can be executed. In this way, by proactively asking follow-up questions, the success rate of user instruction execution can be effectively improved.

[0006] In one possible implementation of the first aspect, the first parameter information includes application information, the first query information includes second query information, the second query information being used to query the user for application information; and receiving the first query information from the second electronic device includes: receiving the second query information from the second electronic device, wherein the second query information is sent to the first electronic device by the second electronic device when it is determined based on the first intent understanding information that the first instruction lacks application information; wherein the first intent understanding information is determined based on the first instruction, and the first intent understanding information includes type information, execution operation information, and first response information, wherein the first response information is used to indicate the first query information.

[0007] In this embodiment, if it is determined that the user's instruction lacks application information, a query can be sent to the user to obtain the missing application information. In this way, by actively asking follow-up questions, the success rate of user instruction execution can be effectively improved.

[0008] Furthermore, the embodiments of this application can understand user commands and obtain parameter information from multiple slots, such as extracting the type information, application information, and execution operation information of the user command, removing redundant information in the user command, and improving the accuracy of subsequent tool recall. In addition, it can generate intelligent follow-up questions to enhance the user experience.

[0009] In one possible implementation of the first aspect, the first intent understanding information is determined based on the first instruction, including: the first intent understanding information is determined based on parameter information corresponding to multiple slots in the first format, the parameter information corresponding to the multiple slots is determined based on the first instruction, and the multiple slots include type information slots, application information slots, execution operation information slots, and response information slots.

[0010] It is understood that, in the embodiments of this application, by designing the first format, redundant information in user instructions can be removed, and colloquial expressions or different sentence structures and tones can be converted into standard instruction-level language, thereby improving the efficiency of subsequent information processing.

[0011] In one possible implementation of the first aspect, the first tool is determined by the second electronic device based on second intent understanding information corresponding to the first instruction and the second instruction. The second intent understanding information includes type information, application information, execution operation information, and second response information.

[0012] In this embodiment, proactive questioning can be achieved, and when the second instruction is obtained, the first instruction can be fused together for further intent understanding. That is, it can fuse multiple rounds of user instructions and perform multiple rounds of intent understanding, thereby improving the accuracy of understanding user instructions and thus improving the recall accuracy of the tool that executes the instructions.

[0013] In one possible implementation of the first aspect, the first tool is determined by the second electronic device based on second intent understanding information corresponding to the first instruction and the second instruction, including: the first tool is determined by the second electronic device based on the second tool, execution operation information in the second intent understanding information, and application information in the second intent understanding information; the second tool is selected by the second electronic device from at least one candidate tool based on the execution operation information in the second intent understanding information and the application information in the second intent understanding information; the at least one candidate tool is determined by the second electronic device based on type information in the second intent understanding information, application information in the second intent understanding information, and a first correspondence relationship, wherein the first correspondence relationship is the correspondence between the type information in the second intent understanding information, the application information in the second intent understanding information, and the at least one candidate tool.

[0014] In the embodiments of this application, the second electronic device can preset the correspondence between different types of information and application information and candidate tools, that is, realize the classification of tool types. In this way, in actual scenarios, candidate tools can be quickly and accurately determined based on application information and type information in intent understanding information, thereby improving tool recall efficiency.

[0015] It is understood that by using the preset type information and different tools under each type information in the embodiments of this application, it is easy to expand. For example, when it is necessary to add new type information in the future, the type information and the corresponding tools can be added directly.

[0016] In addition, the embodiments of this application can accurately match the most suitable tool from the candidate tools by performing operation information and application information, thereby further improving the accuracy of tool recall.

[0017] In one possible implementation of the first aspect, the first query information further includes third query information; and receiving the first query information from the second electronic device further includes: receiving the third query information from the second electronic device, the third query information being sent by the second electronic device to the first electronic device when the second tool determines that the execution operation information lacks second parameter information, the third query information being used to query the user for second parameter information, the second parameter information being parameter information corresponding to the first slot in the second tool; the method further includes: detecting a third instruction from the user, sending a third instruction to the second electronic device, the third instruction being used to indicate the second parameter information; wherein, the first tool is determined by the second electronic device based on the second tool, the execution operation information in the second intent understanding information, and the application information in the second intent understanding information, including: the first tool is determined by the second electronic device based on the second tool, the execution operation information in the second intent understanding information, the application information in the second intent understanding information, and the second parameter information indicated by the third instruction.

[0018] In some embodiments, the first slot may be one or more preset required slots in the second tool, that is, the second parameter information may be parameter information corresponding to one or more required slots. In some embodiments, the required slots of the second tool may have an identifier, for example, the identifier may be "required".

[0019] In this embodiment, when the execution operation information corresponding to a user command lacks the parameter information corresponding to the required slot in the tool, the first electronic device can ask the user follow-up questions to obtain the required parameters. Thus, by actively asking follow-up questions, the success rate of user command execution can be effectively improved. Furthermore, this embodiment can extract slot information from the tool from intent understanding information and add the slot information to the tool, thereby enabling slot extraction and autonomous tool arrangement, improving the flexibility of data processing.

[0020] In one possible implementation of the first aspect, the method further includes: detecting a fourth instruction from the user; and if the type of the fourth instruction does not match a preset type, outputting a first response message indicating that the fourth instruction cannot be executed.

[0021] In this embodiment, if the user instruction does not match the preset type, it can be determined that the user's intent is not related to a shopping scenario, and the user instruction can be left unexecuted. Furthermore, while not executing the user instruction, intelligent reply information can be provided to prompt the user, thus improving the user experience.

[0022] In one possible implementation of the first aspect, the method further includes: detecting a fifth instruction from a user, the fifth instruction indicating multiple tasks; receiving multiple tools from a second electronic device; and performing the multiple tasks based on the multiple tools.

[0023] In this embodiment, user instructions can be decomposed. For example, if a user instruction contains multiple tasks, the user instruction can be decomposed into multiple intent understanding information and the corresponding tools can be obtained respectively, thereby improving tool recall efficiency and further improving the accuracy of user instruction execution.

[0024] Secondly, embodiments of this application provide an information processing method for a second electronic device. The method includes: receiving a first instruction from a first electronic device; if it is determined that the first instruction lacks first parameter information, sending a first query message to the first electronic device, the first query message being used to query a user for first parameter information, the first parameter information including at least one second parameter information from application information and execution operation information; receiving a second instruction from the first electronic device, the second instruction being used to indicate the first parameter information; determining a first tool based on the first instruction and the second instruction, and sending the first tool to the first electronic device, the first tool being used by the first electronic device to perform a first task.

[0025] In one possible implementation of the second aspect, the first parameter information includes application information, the first query information includes second query information, the second query information being used to query the user for application information; and, if it is determined that the first instruction lacks the first parameter information, sending the first query information to the first electronic device includes: determining first intent understanding information based on the first instruction, wherein the first intent understanding information includes type information, execution operation information, and first response information, wherein the first response information is used to instruct the second query information; and if it is determined based on the first intent understanding information that the first instruction lacks application information, sending the second query information to the first electronic device, the second query information being used to query the user for application information.

[0026] In one possible implementation of the second aspect, determining the first intent understanding information based on the first instruction includes: determining parameter information corresponding to multiple slots in the first format based on the first instruction; determining the first intent understanding information based on the parameter information corresponding to multiple slots in the first format, wherein the multiple slots include type information slots, application information slots, execution operation information slots, and response information slots.

[0027] In one possible implementation of the second aspect, determining the first tool based on the first instruction and the second instruction includes: determining second intent understanding information based on the first instruction and the second instruction, the second intent understanding information including type information, application information, execution operation information and second response information; and determining the first tool based on the second intent understanding information.

[0028] In one possible implementation of the second aspect, determining the first tool based on the second intent understanding information includes: determining at least one candidate tool based on type information in the second intent understanding information, application information in the second intent understanding information, and a first correspondence relationship, wherein the first correspondence relationship is the correspondence between the type information in the second intent understanding information, the application information in the second intent understanding information, and the at least one candidate tool; selecting a second tool from the at least one candidate tool based on the execution operation information in the second intent understanding information and the application information in the second intent understanding information; and determining the first tool based on the second tool, the execution operation information in the second intent understanding information, and the application information in the second intent understanding information.

[0029] In one possible implementation of the second aspect, the method further includes: when the execution operation information in the second intent understanding information determined based on the second tool lacks second parameter information, sending a third query message to the first electronic device, the third query message being used to query the user for the second parameter information, the second parameter information being parameter information corresponding to the first slot in the second tool; receiving a third instruction from the first electronic device, the third instruction being used to indicate the second parameter information; determining the first tool based on the second tool, the execution operation information in the second intent understanding information, and the application information in the second intent understanding information, including: determining the first tool based on the second tool, the execution operation information in the second intent understanding information, the application information in the second intent understanding information, and the second parameter information indicated by the third instruction.

[0030] In one possible implementation of the second aspect, the method further includes: receiving a fourth instruction from a first electronic device; and, if the type of the fourth instruction does not match a preset type, sending a first response message to the first electronic device, the first response message indicating that the fourth instruction cannot be executed.

[0031] In one possible implementation of the second aspect, the method further includes: receiving a fifth instruction from a first electronic device, the fifth instruction indicating multiple tasks; determining multiple intent understanding information based on the fifth instruction; determining multiple tools based on the multiple intent understanding information; and sending the multiple tools to the first electronic device for the first electronic device to perform the multiple tasks.

[0032] In one possible implementation of the second aspect, determining the first intent understanding information based on the first instruction includes: determining the first intent understanding information based on the first instruction through a first model.

[0033] In one possible implementation of the second aspect, at least one candidate tool is determined based on type information, application information, and a first correspondence in the second intent understanding information; a second tool is selected from the at least one candidate tool based on execution operation information and application information in the second intent understanding information; and a first tool is determined based on the second tool, execution operation information, and application information in the second intent understanding information. This includes: determining at least one candidate tool using a second model based on type information, application information, and a first correspondence in the second intent understanding information; selecting a second tool from the at least one candidate tool using a third model based on execution operation information and application information in the second intent understanding information; and determining a first tool based on the second tool, execution operation information, and application information in the second intent understanding information.

[0034] Thirdly, embodiments of this application provide an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the method provided by the first electronic device side or the second electronic device side of this application.

[0035] Fourthly, embodiments of this application provide a system including a first electronic device and a second electronic device, wherein the first electronic device is used to execute the method provided by the first electronic device or the second electronic device in the embodiments of this application.

[0036] Fifthly, embodiments of this application provide a computer-readable storage medium storing a program or instructions that, when executed by an electronic device, implement the method on the first electronic device side or the method on the second electronic device side as mentioned in embodiments of this application.

[0037] Sixthly, embodiments of this application provide a computer program product including instructions that, when executed, cause the method on the first electronic device side or the method on the second electronic device side mentioned in embodiments of this application to be implemented.

[0038] In a seventh aspect, embodiments of this application provide a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, such that the chip implements the method on the first electronic device side or the method on the second electronic device side mentioned in embodiments of this application. Attached Figure Description

[0039] Figure 1 According to an embodiment of this application, a schematic diagram of a scenario in which a first electronic device executes a user instruction is shown;

[0040] Figure 2 According to an embodiment of this application, a schematic diagram of a second scenario in which a first electronic device executes a user instruction is shown;

[0041] Figure 3 According to an embodiment of this application, a schematic diagram of a scenario in which a first electronic device performs an information processing method is shown;

[0042] Figure 4 According to an embodiment of this application, a schematic diagram of a scenario in which a second type of first electronic device performs an information processing method is shown;

[0043] Figure 5 According to an embodiment of this application, a schematic diagram of a scenario in which a third type of first electronic device performs an information processing method is shown;

[0044] Figure 6According to an embodiment of this application, a flowchart of a first information processing method is shown;

[0045] Figure 7 According to an embodiment of this application, a schematic diagram of a user instruction is illustrated;

[0046] Figure 8 According to an embodiment of this application, a schematic diagram of a ShareGPT format is illustrated;

[0047] Figure 9 According to an embodiment of this application, a schematic diagram of the system structure is shown;

[0048] Figure 10 According to an embodiment of this application, a schematic diagram illustrates a method for obtaining various types of information and corresponding application information.

[0049] Figure 11 According to the embodiments of this application, a schematic diagram illustrating the process of tool recall in a comparative embodiment and the process of tool recall in the embodiments of this application is shown;

[0050] Figure 12 According to an embodiment of this application, a flowchart illustrating the determination of the first tool is shown;

[0051] Figure 13 According to an embodiment of this application, a flowchart illustrating a second information processing method is shown;

[0052] Figure 14 According to an embodiment of this application, a schematic diagram of the structure of a first electronic device is shown;

[0053] Figure 15 According to an embodiment of this application, a schematic diagram of the structure of a second electronic device is shown. Detailed Implementation

[0054] The illustrative embodiments of this application include, but are not limited to, an information processing method, system, electronic device, storage medium, and program product.

[0055] It should be noted that the specific form of the first electronic device is not limited in the embodiments of this application. The first electronic device provided in the embodiments of this application may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic device may be a mobile phone, smart TV, wearable device, tablet computer, computer with wireless transceiver function, virtual reality (VR) electronic device, augmented reality (AR) electronic device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc. The embodiments of this application do not limit the specific technology or specific device form used in the electronic device. The second electronic device provided in the embodiments of this application may be a cloud server.

[0056] As mentioned earlier, due to the diversity and complexity of user commands, the probability of voice assistants successfully executing user commands is currently low.

[0057] Figure 1 and Figure 2 The diagram illustrates a scenario where the first electronic device executes a user command.

[0058] like Figure 1 As shown, when the voice assistant of the first electronic device 100 detects the user's command "Help me search for my basketball orders", it will output the reply message "Unable to perform this operation!" because the voice assistant cannot determine which application to search for basketball orders.

[0059] like Figure 2 As shown, if the voice assistant of the first electronic device 100 detects the user's command as "Help me buy something in application A", then since the voice assistant cannot determine what to buy, it will also output the reply message "Unable to perform this operation!".

[0060] In other words, there are many situations where the first electronic device cannot execute the user's instructions, resulting in a low probability that the first electronic device will successfully execute the user's instructions.

[0061] To address the aforementioned problems, this application provides an information processing method for a first electronic device. The method includes: detecting a first instruction from a user; if the first instruction lacks first parameter information, outputting first query information, the first query information being used to query the user for first parameter information, the first parameter information including at least one of application information and execution operation information, namely second parameter information (e.g., an execution action such as searching or purchasing, or an execution object such as a product); detecting a second instruction from the user, the second instruction being used to indicate the first parameter information; receiving a first tool from a second electronic device, and performing a first task based on the first tool, wherein the first tool is determined based on the first instruction and the second instruction.

[0062] Based on the above scheme, the first electronic device can proactively ask follow-up questions to obtain the missing information when the user's instruction lacks application information and parameter information (such as the purchased goods) in the execution operation information. Based on the user's reply and the original instruction, the corresponding task can be executed. In this way, by proactively asking follow-up questions, the success rate of user instruction execution can be effectively improved.

[0063] For example, Figure 3 The diagram illustrates a scenario in which a first electronic device performs an information processing method.

[0064] like Figure 3 As shown, when the voice assistant of the first electronic device 100 detects the user's command "Help me search for my basketball orders," and the command lacks application information, the voice assistant will output the follow-up question "Which application do you want to search for my basketball orders?". Furthermore, if the voice assistant detects the user's reply "Application A," it can output the reply: "Okay, I'll do it for you right away," and begin the operation of searching for basketball orders in Application A. This includes actions such as opening Application A, clicking the "All Orders" option in Application A, and entering "basketball" in the search box, to search for basketball orders and display the basketball order page found in Application A.

[0065] In some embodiments, the first electronic device may also display a first card, which may display the task to be executed after the user instruction is parsed and / or the detailed steps of the task to be executed. For example, when the user instruction contains a task, the first card may display the task and the detailed steps of its execution. When the user instruction contains multiple tasks, the first card may display the execution order of the multiple tasks, for example, displaying the multiple tasks in sequence.

[0066] For example, Figure 4 This diagram illustrates a scenario where a first electronic device performs an information processing method. For example... Figure 4As shown, the voice assistant of the first electronic device 100 detects the user's command as "Help me view the reviews of item C in application A," and the first electronic device 100 can output the reply "Okay." Furthermore, the first electronic device 100 can display a first card 001, which may include the task "Go to application A to view product reviews" obtained after parsing the user's command, and the execution details as follows: opening application A, searching for the specified product, the user selecting the desired product, and opening the product reviews.

[0067] For example, Figure 5 This diagram illustrates a scenario where a first electronic device performs an information processing method. For example... Figure 5 As shown, the voice assistant of the first electronic device 100 detects the user's command as "Buy item D in store G of application B, and then check the automatic renewal." The first electronic device 100 can output the reply "Okay." Furthermore, the first electronic device 100 can display a first card 001, which may include the tasks obtained after parsing the user's command: purchasing the item in store G of application B and checking the automatic renewal in application F.

[0068] The information processing method provided in the embodiments of this application will be described in detail below. Figure 6 The schematic diagram illustrates an information processing method according to an embodiment of this application, as shown below. Figure 6 As shown, information processing methods may include:

[0069] 101: The first electronic device detects the user's first command.

[0070] In some embodiments, a user can input a first command through a first application of the first electronic device, such as by voice or by text. That is, the first application of the first electronic device can detect the user's first command.

[0071] In some embodiments, the first application may be a voice assistant, which can be activated by a corresponding wake word or button operation, and the user can input the first command by voice or text.

[0072] For example, the first instruction can be as follows: Figure 3 The example shown is "Help me search my basketball orders." For instance, the first instruction could also be something like... Figure 7 The message displayed reads, "Help me open the review page for app A. It says it's excellent and I've purchased it multiple times."

[0073] 102: The first electronic device sends a first instruction to the second electronic device.

[0074] In some embodiments, after detecting a user's first instruction, the first electronic device may send a first instruction to the second electronic device.

[0075] 103: When the second electronic device determines that the first instruction lacks the first parameter information, it sends a first query message to the first electronic device. The first query message is used to query the user for the first parameter information. The first parameter information includes at least one of the second parameter information in application information and execution operation information.

[0076] In some embodiments, where the first parameter information includes application information, the first query information includes second query information, which is used to query the user for application information. If it is determined that the first instruction lacks the first parameter information, sending the first query information to the first electronic device includes: determining first intent understanding information based on the first instruction, wherein the first intent understanding information includes type information, execution operation information, and first response information, wherein the first response information is used to indicate the second query information. If it is determined based on the first intent understanding information that the first instruction lacks application information, sending the second query information to the first electronic device, the second query information being used to query the user for application information.

[0077] In some embodiments, determining first intent understanding information based on a first instruction includes: determining parameter information corresponding to multiple slots in a first format based on the first instruction, and determining first intent understanding information based on the parameter information corresponding to multiple slots in the first format, wherein the multiple slots include a type information slot, an application information slot, an execution operation information slot, and a response information slot.

[0078] In some embodiments, a first intent understanding information can be determined based on a first instruction using a first model.

[0079] In some embodiments, the type information slot is used to fill in the type information corresponding to the user's instruction. It is understood that the second electronic device may have preset various shopping-related type information. For example, the preset type information may include 12 types: search, open, favorite, review, shopping cart, order, place order, logistics, customer service, invoice, check-in, and launch. After obtaining the user's first instruction, the first instruction can be parsed to determine the type information corresponding to the first instruction. For example, if the user's first instruction is "Help me open item C in application A, its details," then the type information corresponding to this first instruction can be the open type, or the open type in a shopping scenario. If the user's first instruction is "Help me buy a basketball," then the type information corresponding to this first instruction can be the place order type, or the place order type in a shopping scenario.

[0080] In some embodiments, the application information slot is used to fill in the application name indicated by the user command, such as application A. It can be understood that the application information filled in the application information slot can be normalized application information. For example, user commands are diverse; some users habitually refer to application A as application A1, while others habitually refer to application A as application A2. The second electronic device can classify both application A1 and application A2 as application A. For instance, if application A1 is detected in the user command, the content of the application information slot can be filled with application A.

[0081] In some embodiments, the execution operation information slot is used to fill in execution operation information. When the user's first instruction is "Buy me a basketball", the information filled in the execution operation information slot is "Buy a basketball".

[0082] In some embodiments, the slot corresponding to the response information is used to fill in the response output by the first electronic device to the user. For example, when the first instruction does not lack the first parameter information, the response output to the user can be "Done", "Okay", etc. When the first instruction lacks application information, for example, the first instruction is "Help me open item C, its details", the response output to the user can be the second query information, such as "Which application should I use to open the details page of item C?"

[0083] In some embodiments, the first format can be format one: <plan>#Category# Actions in $app$ %Reply%

Task 2

Task 2

[0084] For example, when the first instruction is "Help me open item C in application A, and view its details," the first electronic device can parse the first instruction based on format 1. The parameter information corresponding to the multiple slots obtained can be: the parameter information for the type information slot is "Shopping - Open," the parameter information for the application information slot is "Application A," the parameter information for the execution operation information slot is "Open item C's details," and the parameter information for the response information slot is "Done." The first intent understanding information can be: <plan> #Shopping - Open# Open the details of item B in app A %Done%< / plan> .

[0085] For example, when the first instruction is "Help me open item C, its details", the first electronic device can parse the first instruction based on format 1. The parameter information corresponding to the multiple slots obtained can be: the parameter information for the type information slot is "shopping-open", the parameter information for the application information slot is empty (i.e., the parameter information for the application information slot cannot be obtained), the parameter information for the operation information slot is "open the details of item C", and the parameter information for the response information slot is "Which application should I use to open the details page of item C?" The first intent understanding information can be: <plan> #Shopping - Open# Open item B's details in $$%Which app should I use to open item C's details page%< / plan> .

[0086] In some embodiments, the first format can also be format two: <plan> #Category# Actions within $app$< / plan> <reply> reply< / reply> [Task Two] <plan> <finish>The slots corresponding to the major categories are the type information operations, the slots corresponding to the app are the application information slots, the slots corresponding to the actions are the execution operation information slots, and the slots corresponding to the replies are the response information slots. <plan>Characterizes the beginning of each piece of information. <reply>The response information follows the character. Used to indicate the existence of other tasks, "[Task Two]" refers to the second task, and its format is the same as the first task, i.e., ... The format is the same as before. <finish>Used to indicate the end.

[0087] In some embodiments, the first format can also be format three: <plan> [ {'taskId': 1,'instruction': 'command', 'app': 'app1'}, {

Task 2

Task 3

[0088] It is understandable that in experiments involving multiple tasks, the overall average latency of executing tasks based on Format 1 (e.g., 653ms) is less than that based on Format 2 (e.g., 708ms), and the overall average latency of executing tasks based on Format 2 is less than that based on Format 3 (e.g., 977ms). Format 1 has greater scalability (allowing for flexible addition of content via special symbols) than Format 3 (allowing for the addition of key-value pairs), and Format 1 has greater scalability than Format 2 (allowing for the addition of long string of role identifiers).

[0089] It is understood that, in the embodiments of this application, by designing the first format, redundant information in user instructions can be removed, and colloquial expressions or different sentence structures and tones can be converted into standard instruction-level language, thereby improving the efficiency of subsequent information processing.

[0090] It is understood that in the description of the embodiments of this application in the following embodiments, the first format refers to the above-mentioned format one.

[0091] 104: The first electronic outputs the first query information.

[0092] In some embodiments, when the first electronic device receives a first query message from the second electronic device, it can output the first query message in any way, such as text or voice. For example, it can output the first query message "Which application should I use to open the details page of item C?" via voice.

[0093] 105: The first electronic device detects the user's second instruction.

[0094] In some embodiments, after the first electronic device outputs the first query information, it can detect the user's second instruction. For example, the user's second instruction could be content entered via voice or text, such as "Open the details page of item C in application A".

[0095] 106: The first electronic device sends a second instruction to the second electronic device, the second instruction being used to indicate the first parameter information.

[0096] In some embodiments, after detecting a second instruction from the user, the first electronic device sends a second instruction to the second electronic device. The second instruction may include first parameter information, such as application information.

[0097] In some embodiments, when the second instruction does not include application information, the second electronic device may continue to send query information to the first electronic device after receiving the second instruction in order to query the application information.

[0098] 107: The second electronic device determines the first tool based on the first instruction and the second instruction.

[0099] In some embodiments, determining the first tool based on the first instruction and the second instruction includes:

[0100] Based on the first and second instructions, second intent understanding information is determined. This second intent understanding information includes type information, application information, execution operation information, and a second response information. Based on this second intent understanding information, a first tool is determined.

[0101] In some embodiments, the first instruction and the second instruction can be concatenated and fused based on a first format to obtain second intent understanding information, wherein the format of the second intent understanding information is the same as that of the first intent understanding information, and will not be described here.

[0102] For example, the first instruction is "Help me open item C, its details," and the second instruction is "Open the details page of item C in application A." The first electronic device can then parse the first and second instructions based on a first format. The parameter information corresponding to the multiple slots of the obtained second intent understanding information can be as follows: parameter information corresponding to the type information slot (i.e., type information) "Shopping - Open," parameter information corresponding to the application information slot (i.e., application information) "Application A," parameter information corresponding to the execution operation information slot (i.e., execution operation information) "Open the details of item C," and parameter information corresponding to the response information slot (i.e., second response information) "Done." The second intent understanding information can be: <plan> #Shopping - Open# Open the details of item B in app A %Done%< / plan> .

[0103] It is understood that the first format setting in the embodiments of this application can realize the free splicing of instructions, which is conducive to multi-turn understanding and upstream and downstream interaction. For example, the splicing and fusion method can be to form a dialogue group by taking the user's instructions as input, historical task arrangement (i.e. historical intent understanding information, such as the first intent understanding information) and system response content (i.e. reply information).

[0104] For example, if a user's first instruction is "Help me use photo search," the system's response would be: "Which app would you like to use the photo search function in?" The user's second instruction would be "In app A." The first intent is understood as: <plan> #Shopping-Search# Use the photo search function in $$ %Which app would you like to use the photo search function in?%< / plan> .

[0105] The combined and merged information could be: User: Help me use photo search\nHistory layout: <plan> #Shopping-Search# Use the photo search function in $$ %Which app would you like to use the photo search function in?%< / plan> System reply: Which app would you like to use the photo search function in? User: In app A. The output of the second intent understanding information can be... <plan> #Shopping-Search# Use the photo search function in app A$. Which app would you like to use the photo search function in?< / plan> .

[0106] In some comparative embodiments, the instructions for splicing and merging information can also be in ShareGPT format. Figure 8 The diagram illustrates the ShareGPT format, as shown below. Figure 8 As shown, the ShareGPT format includes roles such as Human (corresponding to human commands), GPT (corresponding to model responses), Observation (corresponding to tool results), and Function Call (corresponding to tool parameters). In this format, Human and Observation must appear in odd-numbered positions, while GPT and Function Call must appear in even-numbered positions. That is, response information only appears in even-numbered output rounds. Therefore, if this format needs to accommodate system response roles, more redundant information needs to be added, leading to increased data processing latency and affecting data processing effectiveness. The first format provided in this application embodiment can encompass response information in each output round, reducing data processing latency.

[0107] In some embodiments, determining a first tool based on second intent understanding information includes: determining at least one candidate tool based on type information in the second intent understanding information, application information in the second intent understanding information, and the correspondence (or first correspondence) between the type information and application information and at least one candidate tool; selecting a second tool from at least one candidate tool based on execution operation information in the second intent understanding information; and determining the first tool based on the second tool and the execution operation information in the second intent understanding information.

[0108] In some embodiments, the electronic device may have a pre-defined correspondence between preset type information and application information (e.g., application name) and at least one candidate tool. In some embodiments, this correspondence may be presented in any form, such as a table or index, and this application embodiment is not limited thereto. After determining the type information in the second intent understanding information, at least one corresponding candidate tool can be determined based on this correspondence.

[0109] For example, the search type information for all applications has 10 corresponding candidate tools, the search type information for application A has 6 corresponding tools, and the search type information for application B has 4 corresponding tools. The opening type information for all applications has 11 corresponding candidate tools, the opening type information for application A has 6 corresponding tools, and the search type information for application B has 5 corresponding tools.

[0110] For example, if the type information in the second intent understanding information is "open type" and the application information is "application A", then six candidate tools can be obtained. These six candidate tools are then matched with the execution operation information in the second intent understanding information to select the second tool that best matches the execution operation information. For example, the keywords in the execution operation information can be matched with the keywords corresponding to each slot in the candidate tool information to obtain the second tool with the highest matching degree.

[0111] In some embodiments, determining the first tool based on the execution operation information in the second tool and the second intent understanding information may include: determining the parameter information corresponding to each slot in the second tool based on the application information in the second intent understanding information and the execution operation information in the second intent understanding information; and filling the parameter information corresponding to each slot into the corresponding slot of the second tool to obtain the first tool.

[0112] For example, the second intention to understand the information is: <plan> #Shopping - Search# Find item C in app A, sorted by price from highest to lowest. %Done%< / plan> The recall of at least one candidate tool includes: a shopping cart content search tool (search_cart_content), a product search tool (search_goods), a search tool in favorite products (search_in_favorite_goods), a search tool in favorite stores (search_in_favorite_stores), an order search tool (search_order), and a store search tool (search_stores). The second tool identified is the product search tool (search_goods), which is represented as search_goods(app=, search_info_slot=, order_type=)'. The product search tool (search_goods) includes three slots, such as an application information slot, a search information slot, and a sorting type slot. Based on the second intent understanding information, the parameter information corresponding to the application information slot is application A, the parameter information corresponding to the search information slot is item C, and the parameter information corresponding to the sorting type slot is price from high to low. The resulting first tool is search_goods(app=application A, search_info_slot=item C, order_type=price from high to low).

[0113] In some embodiments, if the execution operation information in the second intent understanding information determined based on the second tool lacks second parameter information, the second electronic device may send a third query message to the first electronic device. The third query message is used to query the user for the second parameter information, which is the parameter information corresponding to the first slot in the second tool. Furthermore, the second electronic device may receive a third instruction from the first electronic device, which is used to indicate the second parameter information. The first tool is determined based on the second tool, the execution operation information in the second intent understanding information, the application information in the second intent understanding information, and the second parameter information indicated by the third instruction.

[0114] In some embodiments, the first slot may be one or more preset required slots in the second tool, that is, the second parameter information may be parameter information corresponding to one or more required slots. In some embodiments, the required slots of the second tool may have an identifier, for example, the identifier may be "required".

[0115] For example, if the second tool has five mandatory slots, and the parameter information corresponding to four of these slots can be determined based on the execution operation information, then the second parameter information can be the parameter information corresponding to one slot. If the second tool has five mandatory slots, and the parameter information corresponding to three of these slots can be determined based on the execution operation information, then the second parameter information can be the parameter information corresponding to two slots.

[0116] For example, the second intention to understand information is: <plan> #Shopping - Search# Search in app A$ %Done%< / plan> The recalled candidate tools are: a shopping cart content search tool (search_cart_content), a product search tool (search_goods), a search tool in favorites (search_in_favorite_goods), a search tool in favorite stores (search_in_favorite_stores), an order search tool (search_order), and a store search tool (search_stores). The second tool identified is the product search tool (search_goods), which is represented as search_goods(app=, search_info_slot=, order_type=)'. The product search tool (search_goods) includes three slots, such as an application information slot, a search information slot, and a sorting type slot. Based on the second intent understanding information, the parameter information corresponding to the application information slot is application A; the parameter information corresponding to the search information slot is empty, meaning the search information cannot be determined; and the parameter information corresponding to the sorting type slot is also empty. When the search information slot is a required slot and the sorting type slot is a non-required slot, the second electronic device sends a first query message "What are you looking for?" to the first electronic device to inquire about the search information. When the second electronic device receives a third instruction "Item C" from the first electronic device, it can obtain the first tool as search_goods(app=application A, search_info_slot=item C).

[0117] In some embodiments, after obtaining the third instruction, third intent understanding information can be determined based on the first, second, and third instructions, or the third intent understanding information can be determined based on the third instruction and the second intent understanding information. At least one candidate tool can then be determined based on the execution operation information and application information in the third intent understanding information, and a second tool can be determined based on the at least one candidate tool. It is understood that since the third instruction indicates second parameter information, the execution operation information in the third intent understanding information includes the second parameter information. It is also understood that the execution operation information in the third intent understanding information includes the execution operation information and the second parameter information in the second intent understanding information. The application information in the third intent understanding information can be the application information in the second intent understanding information. Based on the execution operation information and application information in the third intent understanding information, the parameter information of each slot in the second tool can be determined.

[0118] It is understood that the tools mentioned in the embodiments of this application can be program code used to perform a certain task. For example, a tool for searching shopping cart contents is program code used to search for contents in a shopping cart.

[0119] In some embodiments, at least one candidate tool can be determined by a second model based on type information in the second intent understanding information, application information in the second intent understanding information, and a first correspondence.

[0120] In some embodiments, a second tool may be selected from at least one candidate tool by a third model based on the execution operation information in the second intent understanding information and the application information in the second intent understanding information, and a first tool may be determined based on the second tool, the execution operation information in the second intent understanding information, and the application information in the second intent understanding information.

[0121] 108: The second electronic device sends the first tool to the first electronic device.

[0122] In some embodiments, after the second electronic device identifies the first tool, it may send the first tool to the first electronic device.

[0123] 109: The first electronic device performs the first task based on the first tool.

[0124] In some embodiments, after receiving the first tool, the first electronic device can perform a first task based on the first tool.

[0125] For example, if a user's first instruction is "Help me open item C, its details," and the second instruction is "Open item C's details page in application A," then the first task can be to open item C's details page in application A, and the first tool can be used to perform this task. For example, the first tool could be used to perform operations such as opening application A's main page, entering item C in the search box on the main page, clicking the search control, clicking item C on the search results page, and entering item C's details page.

[0126] In some embodiments, when the second electronic device determines the intent understanding information based on the user's instruction, if it cannot extract the type information, that is, when the type of the user input instruction does not belong to any of the preset type information, the second electronic device cannot search for the tool. In this case, the second electronic device can send the message that the instruction cannot be executed to the first electronic device so that the first electronic device outputs the message.

[0127] For example, the second electronic device receives a fourth instruction from the first electronic device. If the type of the fourth instruction does not match the preset type, the second electronic device sends a first reply message to the first electronic device. The first reply message indicates that the fourth instruction cannot be executed.

[0128] In some embodiments, when the fourth instruction does not belong to any of the preset types, the type information in the intent understanding information obtained by the first electronic device based on the fourth instruction can be "unknown," and the response information can be "unable to perform the corresponding function." For example, when the fourth instruction is "Open navigation, take me to address E," the obtained intent understanding information is: <plan> #Unknown# Navigation to the nearest bank in $$% Navigation to banks is not currently supported.%< / plan> .

[0129] For example, if the fourth instruction is "Please order shoes for me in app A, pay with app B, and have them delivered to address E, thank you! Then play a song H," the resulting intent understanding information is: <plan>#Shopping - Order Placement# Shoes in app A, pay with app B, delivered to address E. %Done% #Unknown# Playing a song in $$ H% This operation is not currently supported. Try our one-click shopping feature instead!%< / plan> .

[0130] In some embodiments, when a user instruction contains multiple tasks, the second electronic device can acquire multiple intent understanding information, determine multiple tools based on the multiple intent understanding information, and send the multiple tools to the first electronic device, the multiple tools being used by the first electronic device to perform the multiple tasks.

[0131] For example, if a second electronic device receives a fifth instruction from a first electronic device, and the fifth instruction indicates multiple tasks, then it determines multiple intent understanding information based on the fifth instruction, and determines at least one tool based on the multiple intent understanding information; it then sends the at least one tool to the first electronic device, and the at least one tool is used by the first electronic device to perform the multiple tasks.

[0132] It is understandable that among multiple intent understanding information, there may be some intent understanding information that lacks type information. This part of intent understanding information does not require a search tool. In this case, the number of identified tools is less than the number of intent understanding information.

[0133] It is understandable that among multiple intent understanding information, there may be no intent understanding information lacking type information, and the number of identified tools can be equal to the number of intent understanding information.

[0134] For example, if the fifth instruction is "I can actually open the scan function in app A! It's so convenient to view all interaction messages in app B! Oh my god, I can even view my pending orders for the goods I bought in app F!", then the resulting intent understanding information could be: <plan>#Unknown# Enabling the scan function in the Douyin Mall is not currently supported. #Unknown# View all interactive messages in JD.com %This operation is not currently supported% #Shopping - Orders# Viewing my pending orders for purchased items on Xianyu (a second-hand marketplace platform) %Completed%< / plan> This intent understanding information includes three intent understanding pieces, which are obtained through... The information is separated. The type information in the first and second intent understanding information is unknown; therefore, there is no need to search for the corresponding tool, and the information that the operation is not currently supported can be output. Only the first tool needs to be searched for the third intent understanding information.

[0135] In some embodiments, the second electronic device may send multiple tools to the first electronic device at once (e.g., via a list), and the first electronic device may execute multiple tasks sequentially based on the multiple tools. In some embodiments, the multiple tools may also be sent to the first electronic device in multiple installments.

[0136] In some embodiments, after receiving the first instruction, the first electronic device may first identify the intent of the first instruction. Steps 102-109 will only be executed if the intent of the first instruction is a shopping-related intent. For example, after receiving the first instruction, the first electronic device can determine the vertical domain type corresponding to the first instruction. The vertical domain type may include different types such as a lifestyle vertical domain and a weather vertical domain. If it is determined to belong to the lifestyle vertical domain, the intent type of the first instruction is further determined. For example, the intent type may include a shopping intent, a bill payment intent, a delivery intent, and a food delivery intent. When the intent type of the first instruction is determined to be a shopping intent, steps 102-109 are executed. In some embodiments, when the intent of the first instruction is other types of intent, such as a bill payment, delivery, or food delivery intent, the corresponding operation can be executed directly. For example, if the first instruction is to pay the electricity bill, the electronic device can directly execute the electricity bill payment operation.

[0137] Based on the above scheme, the first electronic device can proactively ask follow-up questions to obtain the missing information when the user's instruction lacks application information and parameter information (such as the purchased goods) in the execution operation information. Based on the user's reply and the original instruction, the corresponding task can be executed. In this way, by proactively asking follow-up questions, the success rate of user instruction execution can be effectively improved.

[0138] Furthermore, the embodiments of this application can realize multi-round parsing (e.g., obtaining multi-round intent understanding information) and intelligent decomposition (e.g., decomposing multiple tasks) of complex shopping tasks, thereby achieving efficient execution of user instructions. While reducing user operations, it significantly improves the accuracy and efficiency of shopping task execution.

[0139] The structure of the system mentioned in the embodiments of this application is described below.

[0140] For example, Figure 9 This diagram illustrates the structure of a system provided in an embodiment of this application. The system includes a first electronic device and a second electronic device. The first electronic device may include an intent control module, which may include an intent vertical domain distribution module and a life vertical domain module.

[0141] The second electronic device includes a planning-action (PA) module, an execution agent module, an engineering side, and a large model base platform corresponding to the shopping scenario. The planning-action module includes a planning module, a tool recall module, a tool call (FC) module (or execution module), and a tool definition module.

[0142] In some embodiments, the intent-based vertical domain distribution module receives a user's query, such as a first instruction, and determines the vertical domain type corresponding to the first instruction. The vertical domain type may include different types such as a lifestyle vertical domain and a weather vertical domain. If it is determined that the instruction belongs to a lifestyle vertical domain, the first instruction is sent to the lifestyle vertical domain module.

[0143] The lifestyle vertical module is used to determine the intent type of the first instruction and send the instruction to the corresponding planning module in the second electronic device according to the intent type. For example, the intent type may include shopping intent, bill payment intent, express delivery intent, and food delivery intent. When the intent type of the first instruction is determined to be a shopping intent, the first instruction is sent to the shopping intent planning module in the second electronic device.

[0144] In some embodiments, the planning module can be used for task planning, specifically, it can determine intent understanding information based on user instructions. In some embodiments, the planning module can determine the user's final intent understanding information through multiple rounds of understanding. For example, it can be used to determine first intent understanding information based on a first instruction, wherein the first intent understanding information includes type information, execution operation information, and a first response information, wherein the first response information is used to indicate a second query. If it is determined based on the first intent understanding information that the first instruction lacks application information, a second query is sent to a first electronic device, the second query being used to query the user for application information. And it can determine second intent understanding information based on the first instruction and the second instruction, the second intent understanding information including type information, application information, execution operation information, and a second response information. The application information can be normalized application information. In this case, the second intent understanding information is the final intent understanding information determined by the planning module.

[0145] In some embodiments, the planning module can be used to reject a user instruction when it cannot be determined that the instruction does not belong to any preset type of information, i.e., to issue a rejection message to achieve abnormal interception. For example, the planning module is used to receive a fourth instruction and, if the type of the fourth instruction does not match a preset type, send a first response message to the first electronic device, the first response message indicating that the fourth instruction cannot be executed.

[0146] In some embodiments, the planning module can be used to break down multiple tasks when a user instruction involves multiple tasks. For example, when a fifth instruction is received, indicating multiple tasks, multiple intent understanding information are determined based on the fifth instruction.

[0147] In some embodiments, the planning module can be used to send type information and application information from the intent understanding information corresponding to the user instruction to the tool recall module, such as type information and application information from the second intent understanding information.

[0148] It is understood that the planning module can include a first model, which can also be called a planning model. The functions of the planning module can be executed through the first model. The functions of the first model will be detailed later and will not be repeated here.

[0149] The tool recall module can be used to recall at least one candidate tool based on intent understanding information corresponding to a user command. For example, it can determine a first tool based on second intent understanding information. Specifically, it can be used to determine at least one candidate tool based on type information in the second intent understanding information, application information in the second intent understanding information, and the correspondence between type information, application information, and at least one candidate tool.

[0150] In some embodiments, the tool recall module may send at least one candidate tool to the tool invocation module.

[0151] In some embodiments, the tool recall module may include a second model through which the functions corresponding to the tool recall module can be executed.

[0152] In some embodiments, the tool recall module can recall the tools that rank among the top few (e.g., the top three) in terms of matching degree with the second intent understanding information. In this way, it can be guaranteed that a tool will be recalled, and tools will not be missed.

[0153] The tool definition module can be used to obtain preset type information, as well as the tools corresponding to each preset type and application information. The following section will combine... Figure 10 This paper describes the methods for obtaining various types of information in shopping scenarios and the corresponding tools for each application.

[0154] like Figure 10 As shown, the tool definition module can first obtain a set of instructions by acquiring multiple user instructions within the product scenario scope, for example, the number of instructions in the set can be around 1500. The product scenario scope can refer to the range of scenarios to which the information processing method provided in this embodiment is applicable, such as various applicable tasks (e.g., more than 80 types), various applications (APPs), etc. The multiple user instructions within the product scenario scope can be, for example, common user instructions for performing various tasks in various APPs, such as "Help me buy item C in application A," "Help me search for basketball orders in application B," etc.

[0155] After obtaining the instruction set, high-frequency words corresponding to the instructions (such as search, open, favorite, review, etc.) can be obtained based on similarity algorithms (e.g., vector generation (BAAI General Embedding, BGE) algorithm). These words are then combined with manually defined high-frequency words in the shopping domain to aggregate and generate various types of information (or tool categories). For example, relatively high-frequency type information may include search, open, favorite, review, shopping cart, order, place order, logistics, customer service, invoice, etc. In some embodiments, type information may include more or fewer type information, which is not limited in this application embodiment.

[0156] After obtaining the type information (tool category), we can aggregate and merge the tools that execute user commands for each application under each type of information to obtain the initial version of the tool corresponding to each type of information.

[0157] For example, tools corresponding to search type information could include tools for searching stores, tools for searching products, and tools for searching delivery time. A tool for searching stores can be used to perform store search tasks, a tool for searching products can be used to perform product search tasks, and a tool for searching delivery time can be used to search for item delivery times. For instance, a tool for searching stores could include a tool for searching stores corresponding to application A and a tool for searching stores corresponding to application B.

[0158] For example, if the user command is "to view the delivery time of item C to address E in application A", then the search type information and the search delivery time tool corresponding to application A can be used. This search delivery time tool includes an application information slot (app), a search information slot (search_info), and an address slot (address), and can be represented in the form: asearch_delivery_time(app, search_info, address).

[0159] In some embodiments, a neural network model (such as the Qianwen model) can be used to generalize user commands, generating more user commands (or generalized corpus). The user commands are then processed by a model capable of acquiring type information and tools to obtain output results (i.e., the type information and tools corresponding to the user commands). Based on the correctness of the output results, tools that are easily confused under different types of information are identified. The initial version of type information and tools are then adjusted by merging, deleting, or adding to obtain a complete version of the tools and the final type information. For example, the final type information may include 12 types: search, open, favorite, review, shopping cart, order, place order, logistics, customer service, invoice, check-in, and launch. The complete version of the tools may include 35 tools.

[0160] In some embodiments, the tools corresponding to each type of information can be further subdivided, for example, by application. For instance, the search store tools in the search type information may include one or more search store tools corresponding to application A and one or more search store tools corresponding to application B.

[0161] It is understood that by defining type information and different tools under each type of information in the embodiments of this application, it is easy to expand. For example, when it is necessary to add new type information in the future, the type information and the corresponding tools can be added directly.

[0162] Furthermore, the definition of type information and different tools under each type information in the embodiments of this application can realize coarse and fine classification of the tools required for the instructions, which facilitates the recall and invocation of tools.

[0163] The large model base platform can be used to provide base models and model training data to planning modules and tool calling modules.

[0164] The tool invocation module is used to select a first tool capable of executing user instructions from at least one candidate tool. For example, the tool invocation module can be used to select a second tool from at least one candidate tool based on execution operation information in the second intent understanding information, and to determine the first tool based on the second tool and the execution operation information in the second intent understanding information.

[0165] In some embodiments, the tool invocation module is used to send a third query message to the first electronic device when the execution operation information in the second intent understanding information determined by the second tool lacks the second parameter information. The third query message is used to query the user for the second parameter information, which is the parameter information corresponding to the first slot in the second tool.

[0166] In some embodiments, when a user issues a third instruction based on the third query information, the third instruction can continue to be processed by the intent domain module and then reach the planning module via the life domain module. The planning module can determine the corresponding third intent understanding information (which includes the second parameter information) based on the third instruction and previous instructions, such as the first and second instructions. It then outputs the type and application information from the third intent understanding information to the tool recall module and the application and execution operation information from the third intent understanding information to the tool invocation module. The tool recall module recalls at least one candidate tool based on the type and application information from the third intent understanding information and sends it to the tool invocation module. The tool invocation module determines a second tool based on the application and execution operation information from the third intent understanding information and the at least one candidate tool. Finally, it determines a first tool based on the execution operation information and the second tool.

[0167] In some embodiments, the tool invocation module includes a third model, which may also be referred to as a tool invocation model. The third model enables the functionality of the tool invocation module.

[0168] In some embodiments, the tool invocation module is used to send the tools required to perform the task to the engineering side, such as the first tool.

[0169] The EA module is used to send the response information (e.g., "okay" if the user's instruction can be executed) from the final round of intent understanding information output by the planning module to the first electronic device to enable dialogue interaction with the user. It can be understood that the final round of intent understanding information includes type information and application information, i.e., intent understanding information that does not require further questioning and is not rejection-based, such as the second intent understanding information mentioned above.

[0170] The engineering side can be used to obtain query information from the planning module and the tool invocation module, and send query information to the first electronic device, such as sending second and third query information. It can also receive tools sent by the tool invocation module and send tools to the first electronic device. Furthermore, the engineering side can include an exception handling execution framework for performing exception handling functions, and may also include a Frequently Asked Questions (FAQ) escape route to resolve common questions with preset answers.

[0171] The training of the first model (i.e., the plan model) mentioned in the planning module will be introduced below.

[0172] The training of the first model can be carried out using a two-stage model fine-tuning approach, where the input data used for model training can be a set of user instructions.

[0173] The first stage of training for the first model aims to enable the model to convert input user commands into an initial first format (i.e., meeting the basic layout requirements of the first format), thus training the model's general layout capabilities and output format. During this first stage of training, user command data from open-source datasets (i.e., user command data from various scenarios, excluding shopping scenarios and others) and some business data (i.e., user command data from shopping scenarios) can be used as input data. For example, the initial first format could be: <plan>Actions in $app$ % reply %

Task 2

[0174] The second stage of training for the first model is used to enable the model to convert input data and user commands into a standard first format output. For example, the first format is: <plan>#Category# Actions in $app normalization$ with intelligent responses

Task 2

[0175] It is understood that, in the embodiments of this application, training the first model can output data in the first format, which can effectively improve the recall accuracy of the tool.

[0176] Table 1 illustrates the tool recall accuracy when using different models and different output data formats.

[0177] Table 1

[0178]

[0179] As shown in Table 1, when using the Thousand Questions model and the output data format is shargpt, the total number of user commands used in the experiment was 1931, and the recall tool was correct for 1749 user commands, meaning the tool's recall accuracy reached 90.57%. When using the first model and the output data format is shargpt, the total number of user commands used in the experiment was 1931, and the recall tool was correct for 1808 user commands, meaning the tool's recall accuracy reached 93.63%. When using the Thousand Questions model and the output data format is the first format, the total number of user commands used in the experiment was 1931, and the recall tool was correct for 1821 user commands, meaning the tool's recall accuracy reached 94.30%. When using the first model and the output data format is the first format, the total number of user commands used in the experiment was 1931, and the recall tool was correct for 1836 user commands, meaning the tool's recall accuracy reached 95.08%.

[0180] Specifically, when the first model provided in the embodiments of this application is used, and the output data format is the first format, the tool recall accuracy is the highest. This demonstrates that using the first model and the first output data format in the embodiments of this application can effectively improve the tool recall accuracy.

[0181] It is understood that, in some embodiments, the first model can simultaneously perform the classification task (or category extraction) and the generation task.

[0182] In some embodiments, the classification task may refer to obtaining type information corresponding to the first instruction. Performing a classification task to obtain type information can improve the recall accuracy of the tool.

[0183] For example, Table 2 illustrates the recall accuracy of a recall tool based on type information and a recall tool based on direct user instructions.

[0184] Table 2

[0185]

[0186] As shown in Table 2, when the recall tool is based on type information and instructions that do not belong to the shopping intent are filtered out through engineering, the recall accuracy can reach 97.4%.

[0187] When the tool is directly recalled based on the vector corresponding to the user command, the recall accuracy can reach 94.1%. That is, the embodiments of this application are based on obtaining type information, and recalling tools based on type information can effectively improve the recall accuracy of the tool.

[0188] In some embodiments, generating a task may refer to intelligently rewriting user instructions. In some embodiments, rules can be set in the model's annotation result rules to remove redundant information from the task (e.g., remove titles, exclamations, etc.), rewrite colloquial actions into standard actions, and standardize output (convert inversions, rhetorical questions, etc. into normal order statements), so as to facilitate the model to implement intelligent rewriting functions.

[0189] In some embodiments, the first model can be further enhanced by few-shot inference (i.e., inference with a small number of labeled samples) and live network feedback data (i.e., real user instructions during business use). In some embodiments, complex generalized data can be generated based on open-source large models according to persona, sentence structure, and whether redundant information is added, to expand the model's positive and negative example data, thereby expanding the model's training data and improving the accuracy of model training. For example, the rules for generating complex generalized data may include the following: 1. The generated corpus needs to achieve the same function as the provided task prediction; 2. It should be able to simulate various users, various speaking tones, and various sentence types; 3. Spaces and irrelevant content should be removed from the output; 4. The generated data needs to be different from the provided task prediction data, and simple modifications such as adding or deleting a few words should be avoided as much as possible; 5. The sentence structure and wording should be as generalized and diverse as possible. In some embodiments, the first model may also have rejection recognition capabilities, follow-up questioning capabilities, and intelligent response prediction capabilities.

[0190] In some embodiments, cross-vertical domain data, i.e., data that does not belong to the shopping scenario, such as data on video, travel, and weather, can be added to the model training data. This data is then distributed proportionally across different scenario types (e.g., single / multi-task - cross-vertical domain, multi-turn - segmented domain, etc.). Specifically, scenario types include: single-turn single-task scenarios where the intent is correctly understood; single-turn single-task scenarios where the intent is rejected (i.e., cross-vertical domain, e.g., not belonging to the shopping intent, unable to identify type information); multi-turn single-task scenarios where the intent is correctly understood; multi-turn single-task scenarios requiring follow-up queries on slot parameters; multi-turn single-task scenarios requiring information correction; and multi-turn single-task scenarios requiring segmentation (e.g., tasks need to be segmented, with some tasks representing shopping intent scenarios where type information can be identified, and some tasks representing non-shopping intent (cross-vertical domain) scenarios where type information cannot be identified and need to be rejected). For example, the number of cross-vertical domain data points distributed across the above scenario types can be 300, 30, 80, 30, and 30 respectively. This increases the model's understanding of cross-vertical domain tasks and enables rejection capability. For example, when the type information corresponding to a user instruction does not belong to the preset type, the output type information is "unknown".

[0191] In some embodiments, the first model can also continue valid information normally under multi-turn instructions. For example, if the first instruction is: check the weather at address E, and the second instruction is: buy a basketball from application A and have it delivered there, then the first model can understand that the user's intention is to buy a basketball from application A and have it delivered to address E, and obtain the corresponding intention understanding information and output it.

[0192] In some embodiments, the ability to understand application information can be enhanced by adding data from multiple apps and common user commands for app operations. This enables the model to normalize app names and provide follow-up questions when application information is missing. In some embodiments, follow-up / rejection responses for application information can also be collected and combined with actions and scenarios in user commands, as well as common follow-up responses generated by a large model, to create a generalized intelligent follow-up and rejection corpus. This makes follow-up and rejection responses more intelligent and diverse than fixed response templates. For example, follow-up information in the generalized corpus may include: "Which app do you want to use the photo search function?", "Which app did you take a photo search in?", "Which store did you open the parameters for item C in?", "Please provide the specific app name?", "Which app did you view the order in?", "Which app did you take a photo search in?", and "Please tell me which app you specifically viewed it in?", etc.

[0193] For example, when the user's first instruction is "I want to take a picture and search for this item B", the first intent understanding information generated by the first model can be: <plan> #Shopping - Search# Take a picture of item B in $$ to search for it. Which app should I use for this photo search?< / plan> .

[0194] For example, when a user's first instruction is "Open the details page of item B, then search for item B and sort by price from high to low," the first intent understanding information generated by the first model could be: <plan>#Shopping - Open# Open item B's details page in $$ %Which app should I use to open item B's details page? % #Shopping - Search# To search for item B in $$ and sort by price from highest to lowest, which app should I use?< / plan> As can be seen from the above examples, the response information (i.e., follow-up question information) in the first intent understanding information generated by the first model can take many forms.

[0195] The third model (or tool invocation model, action model) in the tool invocation module provided in the embodiments of this application will be introduced below.

[0196] In this embodiment, the input data of the third model can be at least one candidate tool and intent understanding information (e.g., second intent understanding information). The output data of the third model can be tool information, and the format of the output data can be a DSL format. For example, the format of the output data of the third model can be ['serial number', 'tool name (parameter 1 = parameter value 1, parameter 2 = parameter value 2)']. Wherein, parameter 1 and parameter 2 can refer to the corresponding parameters in the tool, such as slots.

[0197] For example, when there is only one task and the third model outputs a single tool, the output data format can be ['1', 'tool name (parameter 1 = parameter value 1, parameter 2 = parameter value 2)'].

[0198] For example, when there are multiple tasks and the third model outputs multiple tools, the format of the third model output data can be [['1', 'tool name (parameter 1="parameter value 1")'], ['2', 'tool name (parameter 1="parameter value 1", parameter 2="parameter value 2")']]. In this data format, the two tools are independent and do not depend on each other.

[0199] For example, when there are multiple tasks and the third model outputs multiple tools, the format of the third model output data can also be [['1', 'tool name (parameter 1="parameter value 1")'], ['2', 'tool name (parameter 1="result(ID1)", parameter 2="parameter value 2")']]. In this format, the two tools are dependent on each other, that is, the later tool is dependent on the earlier tool.

[0200] In this embodiment of the application, the output format of the second model adopts the DSL format, which has a simple and clear structure. Compared with the JSON format, the DSL format can greatly reduce the number of output characters (tokens) and thus reduce the model response latency while ensuring the same output information. Moreover, the DSL format has better content compatibility.

[0201] For example, Table 3 illustrates the output data of the second model when using DSL and JSON formats.

[0202] Table 3

[0203]

[0204] As shown in Table 3, when using the DSL format, the number of characters (tokens) in the output data is significantly reduced compared to the DSL format.

[0205] In some embodiments, when required parameters are missing in the second tool, concise follow-up questions can be generated based on functions such as `ask_human` and the user's intent. The format of the follow-up questions output by the model can be ['serial number', 'ask_human(inquire=follow-up questions generated by the model)']. For example, when the second tool is a tool for searching in favorite stores (search_in_favorite_stores), if the product name cannot be obtained from the intent understanding information, the output of the third model may include: [['1', 'search_in_favorite_stores(app=application A, search_info_slot=none)'], ['2', 'ask_human(inquire=What is the name of the product you want to search for?)']].

[0206] In some embodiments, the performance of the third model can be further improved. For example, the model error rate can be reduced and the accuracy of tool calls can be improved through a model error case knowledge summarization and feedback mechanism. Frequently Asked Questions (FAQs) can also be used for model knowledge injection and dynamic few-shot guidance (i.e., using a small number of examples to help the model understand task requirements), thereby improving model performance. For example, the data used for knowledge injection may include the original instructions (i.e., the input data to the model) and the intent understanding information data in a standard first format corresponding to the original instructions (i.e., the output data of the model). Furthermore, model stability can be improved by performing engineering validation (validation of function names, parameter names, and closed-domain enumeration values) and App name normalization on the model output results.

[0207] The following examples illustrate the scenarios of finding single-round understanding and multi-round understanding in the embodiments of this application, based on the interaction of the planning module, tool recall module, and tool invocation module in the second electronic device.

[0208] For example, in a single-turn understanding scenario, if the user instruction is "Could you please help me find some items C in application A, sorted by price from highest to lowest?", the intent understanding information output by the planning module would be: <plan> #Shopping - Search# Find item C in app A, sorted by price from highest to lowest. %Done%< / plan> The tool recall module recalls six candidate tools: search_cart_content, search_goods, search_in_favorite_goods, search_in_favorite_stores, search_order, and search_stores. The tool recall module sends these six candidate tools to the tool invocation module. The planning module sends the application information and execution operation information (searching for item C in the application, sorted by price from highest to lowest) from the intent understanding information to the tool invocation module. The second tool identified by the module is the product search tool (search_goods), which is represented as search_goods(app=, search_info_slot=, order_type=). The product search tool (search_goods) includes three slots, such as an application information slot, a search information slot, and a sorting type slot. Based on the second intent understanding information, the parameter information for the application information slot is application A, the parameter information for the search information slot is item C, and the parameter information for the sorting type slot is price from high to low. The resulting first tool is search_goods(app=application A, search_info_slot=item C, order_type=price from high to low). The tool call module outputs information as [['1', 'search_goods(app=application A, search_info_slot=item C, order_type=price from high to low)']].

[0209] For example, in a multi-turn understanding scenario, when the first user instruction received by the planning module is "Find that for me in application A", the first-turn intent understanding information output by the planning module is: <plan> #Shopping - Search# Search in app A$ %Done%< / plan> The six candidate tools recalled are: a tool to search shopping cart content (search_cart_content), a tool to search for products (search_goods), a tool to search among favorited products (search_in_favorite_goods), a tool to search among favorited stores (search_in_favorite_stores), a tool to search for orders (search_order), and a tool to search for stores (search_stores). The tool recall module sends these six candidate tools to the tool invocation module. The planning module sends the application information and execution operation information (search within the application) from the intent understanding information to the tool invocation module. The second tool determined by the tool invocation module is the tool to search for products (search_goods), which is represented as search_goods(app=, search_info_slot=, order_type=). The product search tool (search_goods) includes three slots, such as an application information slot, a search information slot, and a sorting type slot. The parameter information for the application information slot, extracted based on intent understanding, is application A. The parameter information for the search information slot is empty, meaning the search information cannot be determined. The parameter information for the sorting type slot is also empty. When the search information slot is mandatory and the sorting type slot is optional, the tool invocation module sends a first query message, "What are you looking for?", to the first electronic device to request search information.

[0210] When the planning module receives the first round of user instructions as "item C", the second round of intent understanding information output is: <plan> #Shopping - Search# Find item C in app A %Done%< / plan> The planning module outputs the second round of understanding information to the tool recall module, which recalls six candidate tools. The tool recall module then sends these six candidate tools to the tool invocation module. The planning module sends the application information and execution operation information (searching for item C within the application) from the second round of intent understanding information to the tool invocation module. The tool invocation module determines the second tool as the search product tool (search_goods), represented as search_goods(app=, search_info_slot=, order_type=). The search product tool (search_goods) includes three slots, such as an application information slot, a search information slot, and a sorting type slot. Based on the intent understanding information, the parameter information corresponding to the application information slot is application A; the parameter information corresponding to the search information slot is item C (i.e., the search information cannot be determined); and the parameter information corresponding to the sorting type slot is empty. When the sorting type slot is not a mandatory slot, the tool invocation module can obtain the first tool as search_goods(app=application A, search_info_slot=item C). The information output by the tool calling module can be [['1', 'search_goods(app=applicationA, search_info_slot=itemC)']].

[0211] In summary, the planning module provided in this embodiment can achieve functions such as category extraction (i.e., type information extraction), multi-task decomposition, multi-round understanding, intelligent response, and intelligent follow-up questioning. It also supports standardization of apps, actions, and colloquial expressions. The tool definition module, when designing tools, can design application-customized prompt word generation schemes based on prompt word templates, i.e., configuring application-specific parameter enumeration values ​​for different apps (e.g., configuring different slots), dynamically adapting to the differences in parameter values ​​between different apps. Furthermore, the tool definition module can achieve category division, adding or removing tools, and configuring the mapping relationship between tools, type information, and application information. The tool invocation module can extract slot information from the intent understanding information of the tool and add the slot information to the tool, i.e., it can achieve slot extraction and autonomous tool arrangement. It can also achieve functions such as tool invocation, precise tool matching, intelligent response, follow-up questioning, and personal knowledge base querying.

[0212] Figure 11 The illustration shows a process diagram of tool recall in a comparative embodiment and the tool recall process in the embodiment of this application. It is understood that in some comparative embodiments, only the tool invocation module (FC) is used for task planning, i.e., tool recall corresponding to user instructions (single round, multi-round, single task, or multi-task). For example, tools are recalled directly based on the vector corresponding to the user instructions, resulting in low recall accuracy. In the embodiment of this application, the planning module, tool recall module, and action module mentioned above are used together to recall tools corresponding to user instructions. It is understood that the models in the planning module, tool recall module, and action module can adopt the model most suitable for the corresponding function according to the different required execution functions, which can effectively improve the accuracy of tool recall.

[0213] Specifically, the system provided in this application embodiment can collaborate with a large model platform. Based on the large model platform, it can perform two-stage model fine-tuning. Both the planning module and the tool invocation module have the ability to ask follow-up questions and reject tasks. The planning module can initially screen out tasks that are not within its vertical domain and provide intelligent responses, saving the latency required for subsequent processing. Follow-up questions from tool invocation (such as asking about specific products) are passed to the planning module as context for multiple rounds of understanding. In addition, by standardizing the output results of the planning module and the tool invocation module, the overall framework has high scalability. Under conditions that do not require a large amount of data, each model can focus on the task type it is good at, and the labeled input and output facilitate reasonable compatibility and adaptation between the base model and the execution module. In addition, the tool recall module can accurately match major categories of tools and supplement potentially missed scenario tools based on the output of the planning module and the user's original command corpus, thus having higher robustness. Furthermore, through the task orchestration of multi-task / multi-round tasks by the planning module, and the fact that the input of the tool invocation module is a rewritten single-round, single-task scenario, the possibility of model illusion and abnormal output is greatly reduced, and target nodes are also provided for post-processing on the engineering side.

[0214] The system provided in this application embodiment includes a Plan model capable of understanding multi-round and multi-task instructions, possessing the ability to extract major categories, intelligently rewrite, ask follow-up questions, and respond. The Action model extracts complex multi-slot parameters, acquires multiple tools, and has the ability to proactively ask follow-up questions and dynamically complete parameters. Combining the Plan-Action dual-model approach (i.e., the Plan model and the Action model) forms a complete PA system, enabling the orchestration and decomposition of complex tasks, improving task execution efficiency and accuracy.

[0215] The steps for determining the first tool mentioned in step 107 will be briefly described below, taking into account the structure of the system provided in the embodiments of this application. Figure 12 The flowchart illustrating the process of determining the first tool is shown. For example... Figure 12 As shown, the method may include:

[0216] 201: The planning module sends application information and type information to the tool recall module.

[0217] In some embodiments, the planning module may send application information and type information from intent understanding information (e.g., second intent understanding information) to the tool recall module.

[0218] 202: The tool recall module sends a tool list of at least one candidate tool to the tool invocation module.

[0219] In some embodiments, the tool recall module may determine at least one candidate tool based on type information, application information, and the correspondence between type information, application information and at least one candidate tool, and send a tool list of at least one candidate tool to the tool invocation module.

[0220] 203: The planning module sends task description information to the tool invocation module.

[0221] In some embodiments, the planning module sends task description information to the tool invocation module. This task description information may include application information and execution operation information from the intent understanding information.

[0222] 204: The tool call module determines that there are missing parameters and sends a follow-up message to the first electronic device.

[0223] In some embodiments, the tool invocation module may determine that a lack of parameters is missing and output follow-up information to the first electronic device. This may include the tool invocation module determining a second tool from a candidate list of at least one candidate tool, and sending a third query (i.e., follow-up information) to the first electronic device if the execution operation information in the second intent understanding information determined based on the second tool lacks second parameter information.

[0224] 205: The tool calling module sends DSL format tool orchestrations downstream.

[0225] In some embodiments, the tool invocation module can send a tool arrangement in DSL format to downstream (e.g., the engineering side), wherein the tool arrangement may refer to the tool information output by the aforementioned third model, for example, the format may be ['serial number', 'tool name (parameter 1=parameter value 1, parameter 2=parameter value 2)'].

[0226] It is understood that, in some embodiments, the first electronic device in the system mentioned in the embodiments of this application, in addition to Figure 9 The structure shown may also include business plugins and PA software development kits (SDKs). The engineering side in the second electronic device may also be called AgentAbilityService or PA cloud-side microservices, and the tool recall module may also be called an open platform.

[0227] The information processing method provided in this application will be further described below with reference to the system structure provided in the embodiments of this application. Figure 13 The diagram illustrates a flowchart of an information processing method, wherein step 301 is a further description of step 101, steps 302-305 are further descriptions of step 102, steps 303-320 are further descriptions of steps 103 to 107, and steps 321-325 are further descriptions of step 109. The method may include:

[0228] 301: The intent control module receives user commands (queries).

[0229] For example, the user instruction could be the first instruction mentioned above.

[0230] 302: The intent control module distributes intents, sending user commands to the corresponding business plugins.

[0231] It is understandable that the intent control module can determine the intent corresponding to a user command and send the user command to the corresponding business plugin based on that intent. For example, if the intent corresponding to a user command is a shopping intent, then the user command will be sent to the business plugin corresponding to shopping.

[0232] 303: The business plugin sends user instructions to the PA SDK.

[0233] In some embodiments, after the business plugin obtains the user instruction, it sends the user instruction to the PA SDK.

[0234] 304: PA SDK obtains context and state.

[0235] In some embodiments, the PA SDK can obtain the current state of the electronic device and the context corresponding to the user instruction, such as previous instructions, and obtain the current state information. The state information may indicate whether the electronic device currently has the application corresponding to the user instruction installed. If the application corresponding to the user instruction is installed or the application corresponding to the user instruction cannot be identified, subsequent steps are executed. If the application corresponding to the user instruction is not installed, subsequent steps are not executed.

[0236] It is understood that after step 304, the steps of tool disassembly, returning the tool list, disassembly result callback, and executing tasks based on the disassembly result (i.e., the first tool) can be performed. The implementation of tool disassembly and returning the tool list is described below based on steps 305-324, and the disassembly result callback is described based on step 325.

[0237] 305: The PA SDK sends user commands and context to the proxy capability service.

[0238] 306: Pre-processing for proxy capability services.

[0239] In some embodiments, the agent capability service can perform preprocessing, such as prompt assembly, to obtain processed user instructions. It is understood that user instructions are generally colloquial or fragmented. Prompt assembly can remove redundant information and piece together context, ensuring the processed instructions conform to a preset template, including a subject, predicate, and object, but excluding interjections. For example, if the user instruction is "Can you help me buy a basketball?", the instruction obtained through prompt assembly could be "Help me buy a basketball".

[0240] 307: The proxy capability service sends a planning request to the planning module.

[0241] In some embodiments, the agent capability service sends a planning request to the planning module (or plan model) so that the planning module can plan the user instruction, i.e., obtain the first intent understanding information corresponding to the user instruction. The planning request may include the pre-processed user instruction.

[0242] 308: The planning module determines that the user instruction lacks an application name and sends follow-up information to the proxy capability service.

[0243] In some embodiments, the planning module determines that the user instruction (i.e., the pre-processed instruction) lacks an application name and sends follow-up information (i.e., second inquiry information) to the agent capability service. For example, the planning module may obtain first intent understanding information based on the user instruction (i.e., the pre-processed instruction), wherein the first intent understanding information includes type information, execution operation information, and first response information, wherein the first response information is used to indicate the second inquiry information. If it is determined based on the first intent understanding information that the user instruction lacks application information, the second inquiry information is sent to the first electronic device, the second inquiry information being used to inquire about the user's application information.

[0244] 309: The proxy capability service sends follow-up questions to the PA SDK.

[0245] In some embodiments, the agent capability service receives follow-up information (second inquiry information) sent by the planning module and can send follow-up information (second inquiry information) to the PA SDK.

[0246] 310: Business plugins retrieve follow-up information from the PA SDK callback.

[0247] In some embodiments, after the PA SDK obtains the follow-up information (second inquiry information), it can send the follow-up information to the business plugin. In some embodiments, the business plugin can retrieve the follow-up information (second inquiry information) from the PA SDK through callback functions or other means.

[0248] 311: The business plugin performs the on-screen operation.

[0249] In some embodiments, after receiving the follow-up question (second inquiry information), the service plugin can perform a screen display operation, that is, control the electronic device to output (display) the follow-up question (second inquiry information).

[0250] It is understandable that after the user issues a second instruction based on the follow-up information, the process can proceed to step 301, for example, steps 301-307. This allows the planning module to obtain the user's second instruction (i.e., the pre-processed second instruction).

[0251] 312: The planning module returns the task list to the agent capability service.

[0252] In some embodiments, after receiving the second instruction, the planning module can determine the second intent understanding information based on the first and second instructions, and then send the task list, which includes the second intent understanding information, to the agent capability service.

[0253] In some embodiments, when the planning module determines multiple intent understandings based on user instructions, i.e., when the user instructions contain multiple tasks, the task list may include multiple intent understanding information.

[0254] 313: The proxy capability service sends a request to the open platform to recall the toolset.

[0255] In some embodiments, when the agent capability service receives a task list sent by the planning module, such as a task list that includes second intent understanding information, it can send a recall toolkit request to the open platform, wherein the recall toolkit request includes application information and type information from the second intent understanding information.

[0256] It is understood that in some embodiments, when the task list includes multiple intent understanding information, the agent capability service can send multiple recall tool set requests to the open platform in sequence, and each recall tool request contains application information and type information from the corresponding intent understanding information.

[0257] In some embodiments, when the task list includes multiple intent understanding information, the agent capability service can send a recall toolkit request to the open platform. The recall toolkit request includes application information and type information from the multiple intent understanding information.

[0258] The following section will use the application information and type information in the second intent understanding information included in the recall toolset request as an example for further explanation.

[0259] 314: The open platform returns toolsets to the proxy capability service.

[0260] In some embodiments, after receiving a request to recall toolset, the open platform can determine at least one candidate tool based on the type information in the second intent understanding information, the application information in the second intent understanding information, and the correspondence between the type information and the application information and at least one candidate tool, and send at least one candidate tool (e.g., the top 10 candidate tools in matching degree ranking) to the proxy service capability, i.e., toolset.

[0261] 315: The proxy capability service sends a tool disassembly request to the tool invocation (FC) module.

[0262] In some embodiments, after receiving at least one candidate tool, the proxy capability service sends a tool disassembly request to the tool invocation module (or FC model), wherein the tool disassembly request includes at least one candidate tool, application information in the second intent understanding information, and execution operation information in the second intent understanding information.

[0263] 316: The tool call module determines that a required parameter is missing and sends follow-up information to the proxy capability service.

[0264] In some embodiments, when the tool invocation module determines that a required parameter is missing, sending follow-up information to the proxy capability service may include: the tool invocation module selecting a second tool from at least one candidate tool based on the execution operation information in the second intent understanding information; and when it is determined based on the second tool that the execution operation information in the second intent understanding information lacks second parameter information, sending follow-up information (i.e., third inquiry information) to the proxy capability service. The third inquiry information is used to ask the user for the second parameter information, which is the parameter information corresponding to the first slot in the second tool.

[0265] In some embodiments, when the tool invocation module does not lack parameters in the execution operation information in the second intent understanding information determined by the second tool, it generates the first tool based on the application information and execution operation information in the second intent understanding information and the second tool.

[0266] 317: The proxy capability service sends follow-up questions to the PA SDK.

[0267] In some embodiments, the agent capability service receives follow-up information (third inquiry information) sent by the planning module and can send follow-up information (third inquiry information) to the PA SDK.

[0268] 318: Business plugins retrieve follow-up information from the PA SDK callback.

[0269] In some embodiments, after obtaining the follow-up information (third inquiry information), the PA SDK can send the follow-up information to the business plugin. In some embodiments, the business plugin can retrieve the follow-up information (third inquiry information) from the PA SDK through callback functions or other means.

[0270] 319: Business plugins perform on-screen operations.

[0271] In some embodiments, after receiving follow-up questions (third inquiry information), the business plugin can perform a screen display operation, that is, control the electronic device to output (display) the follow-up questions (third inquiry information).

[0272] 320: The tool invocation module sends a list of tools to the proxy capability service.

[0273] In some embodiments, when the user issues a third instruction again based on the third query information, the process can proceed to step 301. The intent control module can detect the third instruction and continue executing steps 302-307, enabling the planning module to receive the planning request. Based on the pre-processed third instruction and previous instructions, such as the first and second instructions, the module determines the corresponding third intent understanding information (which includes the second parameter information). Steps 312-315 are then executed. The tool invocation module determines the second tool based on the execution operation information in the third intent understanding information and at least one candidate tool. The first tool is determined based on the application information, the execution operation information, and the second tool.

[0274] After identifying the first tool, the tool invocation module can send a tool list, i.e., tool information, to the proxy capability service. This tool information includes the first tool. It can be understood that the tool information is in the aforementioned DSL format.

[0275] 321: Post-processing of proxy capability services.

[0276] In some embodiments, after receiving the tool list, the proxy capability service can perform post-processing on the tool list, such as normalizing the APP names contained in the tools in the tool list.

[0277] 322: The proxy capability service sends the results to the PA SDK.

[0278] In some embodiments, the proxy capability service can obtain the result of post-processing the tool list, i.e., the post-processed tool list, and send it to the PA SDK.

[0279] 323: PA SDK parses DSL and orchestrates tasks.

[0280] In some embodiments, after receiving the post-processed tool list, the PA SDK can parse the tool list (i.e., tool information in DSL format) and orchestrate tasks, thus obtaining the orchestrated tool list. For example, if the tool information includes one tool, the orchestrated tool list will include one tool. In scenarios where the tool information includes multiple tools, the orchestrated tool list will include multiple tools arranged in sequence.

[0281] 324: The PA SDK performs state storage.

[0282] In some embodiments, PA SDK state storage may refer to storing the current list of tools.

[0283] 325: List of callback tools from PA SDK for business plugins.

[0284] In some embodiments, the PA SDK can send a tool list to a business plugin. In some embodiments, the business plugin can retrieve the tool list from the PA SDK via callback functions or other methods.

[0285] It is understandable that shopping scenarios are highly complex, involving numerous instructions and tools. Especially in shopping scenarios with multiple categories, multiple conditions, and multiple rounds of interaction, users often need to frequently switch search conditions, compare different products, and call multiple tools (such as price comparison, inventory check, coupon redemption, etc.), which consumes a lot of time and is prone to incomplete information or improper task breakdown, which can affect the execution of the final instructions.

[0286] Based on the above scheme, the first electronic device can proactively ask follow-up questions to obtain the missing information when the user's instruction lacks application information and parameter information (such as the purchased goods) in the execution operation information. Based on the user's reply and the original instruction, the corresponding task can be executed. In this way, by proactively asking follow-up questions, the success rate of user instruction execution can be effectively improved.

[0287] Furthermore, the embodiments of this application employ a layered design of planning (e.g., acquiring intent understanding information) and execution (e.g., recall tools) to reduce task coupling, thereby improving system scalability and stability. This enables multi-round parsing, intelligent decomposition, and efficient execution of complex shopping tasks, significantly improving the accuracy and efficiency of shopping decisions while reducing user operations.

[0288] The hardware structure of the first electronic device will be described below. Figure 14 A schematic diagram of the structure of a first electronic device 100 is shown. For example... Figure 14 As shown, the first electronic device 100 includes a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, buttons 101, and a display screen 102, etc.

[0289] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the first electronic device. In other embodiments of this application, the first electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0290] Processor 110 may include one or more processing units, such as processing modules or circuits of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Micro-programmed Control Unit (MCU), Artificial Intelligence (AI) processor, or Field Programmable Gate Array (FPGA). Different processing units may be independent devices or integrated into one or more processors. Processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in processor 110 is a cache memory 180.

[0291] It is understood that the processor 110 can be used to execute the steps on the first electronic device side of the information processing method mentioned in the embodiments of this application.

[0292] The power module 140 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component manages the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module receives charging input from a charger; the power management module connects to the power supply and the processor 110. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 110, the display 102, the camera 170, and the wireless communication module 120, etc.

[0293] The wireless communication module 120 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 120 can provide solutions for wireless communication applications on the mobile phone 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near-field communication (NFC), and infrared (IR) technologies. The mobile phone 10 can communicate with networks and other devices through wireless communication technologies.

[0294] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The sensor module 190 may include proximity sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0295] The audio module 150 is used to convert digital audio information into analog audio signals for output, or to convert analog audio input into digital audio signals. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 may be located in the processor 110, or some functional modules of the audio module 150 may be located in the processor 110. In some embodiments, the audio module 150 may include a speaker, a handset, a microphone, and a headphone jack.

[0296] The hardware structure of the second electronic device will be described below. Figure 15 A schematic diagram of the structure of a second electronic device 200 is shown. For example... Figure 15 As shown, the second electronic device 200 may include one or more processors 201, one or more memories 202, and one or more communication interfaces 203. The one or more processors 201, one or more memories 202, and one or more communication interfaces 203 are coupled to each other. In some embodiments, the one or more memories 202 may be used to store instructions executed by the one or more processors 201, or to store input data required for the one or more processors 201 to execute instructions, or to store data generated after the one or more processors 201 executes instructions. Optionally, see [link to documentation]. Figure 2 One or more processors 201, one or more memories 202, and one or more communication interfaces 203 are interconnected via a bus 204.

[0297] The processor 201 in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor. When at least one instruction in the processor 201 executes, the steps of the second electronic device 200 mentioned in the embodiments of this application can be implemented.

[0298] This application provides an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the method of the first electronic device side or the second electronic device side in this application.

[0299] This application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a first electronic device, the method of the first electronic device in this application is implemented; or when the program or instructions are executed by a second electronic device, the method of the second electronic device in this application is implemented.

[0300] This application provides a computer program product, including instructions that, when executed, cause the method on the first electronic device side or the method on the second electronic device side in this application to be implemented.

[0301] This application provides a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, such that the chip implements the method on the first electronic device side of this application, or the method on the second electronic device side of this application.

[0302] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0303] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0304] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0305] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on or on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, read-only memory, magneto-optical disks, random access memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0306] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0307] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0308] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0309] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.< / plan> < / plan> < / finish> < / plan> < / plan> < / plan>

Claims

1. An information processing method, characterized in that, For a first electronic device, the method includes: Upon detecting the user's first instruction, the first instruction is sent to the second electronic device; The system receives a first query message from the second electronic device and outputs the first query message. The first query message is sent by the second electronic device to the first electronic device when it determines that the first instruction lacks first parameter information. The first query message is used to query the user for the first parameter information. The first parameter information includes at least one of the second parameter information in application information and execution operation information. Upon detecting a second instruction from the user, the second electronic device sends the second instruction, which is used to indicate the first parameter information; Receive a first tool from the second electronic device and perform a first task based on the first tool, wherein the first tool is determined by the second electronic device based on the first instruction and the second instruction; When the first parameter information includes the application information, the first query information includes second query information, which is used to query the user for the application information, and... The receipt of the first query information from the second electronic device includes: Receive the second query information from the second electronic device, wherein the second query information is sent by the second electronic device to the first electronic device when it determines, based on the first intent understanding information, that the first instruction lacks the application information; The first intent understanding information is determined based on the first instruction. The first intent understanding information includes type information, execution operation information, and first response information, wherein the first response information is used to indicate the first query information.

2. The information processing method according to claim 1, characterized in that, The first intent understanding information is determined based on the first instruction and includes: The first intent understanding information is determined based on parameter information corresponding to multiple slots in the first format. The parameter information corresponding to the multiple slots is determined based on the first instruction. The multiple slots include type information slots, application information slots, execution operation information slots, and response information slots.

3. The information processing method according to claim 2, characterized in that, The first tool is determined by the second electronic device based on the second intent understanding information corresponding to the first instruction and the second instruction. The second intent understanding information includes the type information, the application information, the execution operation information, and the second response information.

4. The information processing method according to claim 3, characterized in that, The first tool is determined by the second electronic device based on second intent understanding information corresponding to the first instruction and the second instruction, including: The first tool is determined by the second electronic device based on the second tool, the execution operation information in the second intent understanding information, and the application information in the second intent understanding information. The second tool is selected by the second electronic device from at least one candidate tool based on the execution operation information in the second intent understanding information and the application information in the second intent understanding information. The at least one candidate tool is determined by the second electronic device based on the type information in the second intent understanding information, the application information in the second intent understanding information, and a first correspondence relationship. The first correspondence relationship is the correspondence between the type information in the second intent understanding information, the application information in the second intent understanding information, and the at least one candidate tool.

5. The information processing method according to claim 4, characterized in that, The first query information also includes a third query information; and, The receiving of the first query information from the second electronic device further includes: The system receives a third query message from the second electronic device. The third query message is sent by the second electronic device to the first electronic device when the second tool determines that the execution operation information lacks the second parameter information. The third query message is used to query the user for the second parameter information, which is the parameter information corresponding to the first slot in the second tool. The method further includes: Upon detecting a third instruction from the user, the third instruction is sent to the second electronic device, the third instruction being used to indicate the second parameter information; The first tool is determined by the second electronic device based on the second tool, the execution operation information in the second intent understanding information, and the application information in the second intent understanding information, including: the first tool is determined by the second electronic device based on the second tool, the execution operation information in the second intent understanding information, the application information in the second intent understanding information, and the second parameter information indicated by the third instruction.

6. The information processing method according to any one of claims 1-5, characterized in that, The method further includes: The user's fourth command was detected; If the type of the fourth instruction does not match the preset type, a first response message is output, which indicates that the fourth instruction cannot be executed.

7. The information processing method according to any one of claims 1-5, characterized in that, The method further includes: A fifth instruction from the user was detected, which instructs multiple tasks; Receive multiple tools from a second electronic device; The multiple tasks are performed based on the multiple tools.

8. An information processing method, characterized in that, For a second electronic device, the method includes: Receive a first instruction from the first electronic device; If it is determined that the first instruction lacks the first parameter information, a first query message is sent to the first electronic device. The first query message is used to query the user for the first parameter information. The first parameter information includes at least one of the second parameter information in application information and execution operation information. Receive a second instruction from the first electronic device, the second instruction being used to indicate the first parameter information; A first tool is determined based on the first instruction and the second instruction, and the first tool is sent to the first electronic device, wherein the first tool is used by the first electronic device to perform a first task; When the first parameter information includes the application information, the first query information includes second query information, which is used to query the user for the application information; and... The step of sending a first query message to the first electronic device when it is determined that the first instruction lacks the first parameter information includes: First intent understanding information is determined based on the first instruction, wherein the first intent understanding information includes type information, execution operation information and first response information, wherein the first response information is used to indicate the second query information; If, based on the first intent understanding information, it is determined that the first instruction lacks the application information, the second query information is sent to the first electronic device, the second query information being used to query the user for the application information.

9. The information processing method according to claim 8, characterized in that, The determination of the first intent understanding information based on the first instruction includes: Based on the first instruction, determine the parameter information corresponding to multiple slots in the first format; The first intent understanding information is determined based on the parameter information corresponding to the multiple slots in the first format, wherein the multiple slots include type information slots, application information slots, execution operation information slots, and response information slots.

10. The information processing method according to claim 9, characterized in that, The step of determining the first tool based on the first instruction and the second instruction includes: Based on the first instruction and the second instruction, second intent understanding information is determined, the second intent understanding information including the type information, the application information, the execution operation information, and the second response information; The first tool is determined based on the second intent understanding information.

11. The information processing method according to claim 10, characterized in that, The step of determining the first tool based on the second intent understanding information includes: At least one candidate tool is determined based on the type information in the second intent understanding information, the application information in the second intent information, and the first correspondence, wherein the first correspondence is the correspondence between the type information in the second intent understanding information, the application information in the second intent understanding information, and at least one of the candidate tools; A second tool is selected from at least one candidate tool based on the execution operation information and the application information in the second intent understanding information; The first tool is determined based on the second tool, the execution operation information in the second intent understanding information, and the application information in the second intent understanding information.

12. The information processing method according to claim 11, characterized in that, The method further includes; If, based on the second tool, it is determined that the execution operation information in the second intent understanding information lacks the second parameter information, a third query message is sent to the first electronic device. The third query message is used to ask the user for the second parameter information, which is the parameter information corresponding to the first slot in the second tool. Receive a third instruction from a first electronic device, the third instruction being used to indicate the second parameter information; The step of determining the first tool based on the second tool, the execution operation information in the second intent understanding information, and the application information in the second intent understanding information includes: The first tool is determined based on the second tool, the execution operation information in the second intent understanding information, the application information in the second intent understanding information, and the second parameter information indicated by the third instruction.

13. The information processing method according to claim 8, characterized in that, The method further includes: Receive a fourth instruction from the first electronic device; If the type of the fourth instruction does not match the preset type, a first response message is sent to the first electronic device, the first response message indicating that the fourth instruction cannot be executed.

14. The information processing method according to claim 8, characterized in that, The method further includes: Receive a fifth instruction from the first electronic device, the fifth instruction indicating multiple tasks; Based on the fifth instruction, multiple intent understanding information are determined; Multiple tools are determined based on the aforementioned multiple intent understanding information; The plurality of tools are sent to the first electronic device, the plurality of tools being used by the first electronic device to perform the plurality of tasks.

15. The information processing method according to claim 8, characterized in that, Determining the first intent understanding information based on the first instruction includes: determining the first intent understanding information based on the first instruction using a first model.

16. The information processing method according to claim 11, characterized in that, The process of determining at least one candidate tool based on type information in the second intent understanding information, application information in the second intent understanding information, and a first correspondence, selecting a second tool from the at least one candidate tool based on execution operation information and application information in the second intent understanding information, and determining the first tool based on the second tool, execution operation information, and application information in the second intent understanding information includes: At least one candidate tool is determined by the second model based on the type information in the second intent understanding information, the application information in the second intent understanding information, and the first correspondence; A second tool is selected from at least one candidate tool using a third model based on the execution operation information and application information in the second intent understanding information, and the first tool is determined based on the second tool, the execution operation information and application information in the second intent understanding information.

17. An electronic device, characterized in that, include: One or more processors; One or more memories; the one or more memories storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1-7 or the method of any one of claims 8-16.

18. A system, characterized in that, It includes a first electronic device and a second electronic device, wherein the first electronic device is used to perform the method of any one of claims 1-7, and the second electronic device is used to perform the method of any one of claims 8-16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by the first electronic device, implement the method of any one of claims 1-7 or the method of any one of claims 8-16.

20. A computer program product, characterized in that, Includes instructions that, when executed, cause the method of any one of claims 1-7 or the method of any one of claims 8-16 to be implemented.

21. A chip, characterized in that, The chip includes a processor coupled to a memory for executing a computer program or instructions stored in the memory, such that the chip implements the method of any one of claims 1-7 or the method of any one of claims 8-16.

Images