Large language model task processing method and apparatus, electronic device, and medium

Abstract

The present disclosure provides a large language model task processing method, device, electronic equipment and medium. The large language model task processing method comprises: in response to a task action on a target interface, obtaining an environment description component corresponding to the target interface and tool component information of the target interface from a storage hierarchy, the storage hierarchy storing the environment description component and the tool component information according to interfaces; injecting, through a knowledge injection component, the environment description component corresponding to the target interface and the tool component information into a large language model, so that the large language model generates a tool component calling instruction corresponding to the task action according to environment information in the environment description component and the tool component information; and executing the tool component calling instruction to execute the task action. The embodiments of the present disclosure can improve the accuracy of task processing while maintaining the efficiency of task processing of the large language model. The embodiments of the present disclosure can be applied to various application program task processing scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a method, apparatus, electronic device, and medium for processing large language model tasks. Background Technology

[0002] Currently, while large language models continue to make breakthroughs and can be generalized to various domains, there has been no substantial progress in tasks requiring user interface manipulation. This is because web interfaces consist of a large amount of HTML, CSS, JavaScript, text, images, and videos. Users only see elements like text, images, and videos, but the HTML, CSS, and JavaScript actually occupy the vast majority of the page's characters. Manipulating such interfaces requires finding the necessary key characters within a large amount of meaningless text. Large language models can only handle limited context length, and due to the limitations of attention mechanisms, excessively long contexts can lead to partial memory loss, resulting in serious problems such as decreased response quality.

[0003] To address this issue, one related technique cleanses the interface data using methods such as regular expression matching. While this reduces the context length, it can lead to the loss of crucial information, reducing the accuracy of large language model processing tasks. Another related technique acquires tool component information and background information from the interface by simulating element clicks. For example, tool components are identified within the interface's HTML / CSS / JS / text / images / videos. By actually clicking on these tool components, information about the component and its execution environment is obtained, providing accurate tool component and environment information for large language model processing tasks. However, actually clicking on each tool component results in low task processing efficiency. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and medium for processing large language model tasks, which can improve the accuracy of task processing while maintaining the efficiency of large language model task processing.

[0005] According to one aspect of this disclosure, a method for processing large language model tasks is provided, comprising:

[0006] In response to a task action on a target interface, the environment description component corresponding to the target interface and the tool component information of the target interface are obtained from the storage hierarchy. The storage hierarchy stores the environment description component and the tool component information by interface. The environment description component contains environment information when calling each tool component in the interface.

[0007] The knowledge injection component injects the environment description component and tool component information corresponding to the target interface into the large language model, so that the large language model generates the tool component invocation instruction corresponding to the task action based on the environment information in the environment description component and the tool component information.

[0008] Execute the tool component invocation instruction to perform the task action.

[0009] According to one aspect of this disclosure, a large language model task processing apparatus is provided, comprising:

[0010] The first acquisition unit is configured to, in response to a task action on the target interface, acquire from the storage hierarchy the environment description component corresponding to the target interface and the tool component information of the target interface, wherein the storage hierarchy stores the environment description component and the tool component information by interface, and the environment description component contains environment information when calling each tool component in the interface;

[0011] An injection unit is used to inject the environment description component and the tool component information corresponding to the target interface into the large language model through a knowledge injection component, so that the large language model generates a tool component invocation instruction corresponding to the task action based on the environment information in the environment description component and the tool component information.

[0012] An execution unit is used to execute the tool component invocation instructions to perform the task action.

[0013] Optionally, the large language model task processing device further includes:

[0014] The second acquisition unit is used to acquire an interface sequence, the interface sequence including multiple consecutive interfaces obtained from the start interface through a series of control triggers, and the multiple consecutive interfaces including the target interface.

[0015] The third acquisition unit is used to acquire, for each interface in the interface sequence, the tool component information of the interface and the environment information when calling each tool component in the interface;

[0016] The first generation unit is used to generate the environment description component based on the environment information when calling each tool component in the interface;

[0017] A storage unit is used to store the environment description component and tool component information of the interface according to the position of the interface in the interface training, so as to generate the storage hierarchy structure.

[0018] Optionally, the large language model task processing device further includes:

[0019] The receiving unit is used to receive task instructions;

[0020] The second generation unit is used to generate a task plan based on the task instructions, the task plan including a plurality of task actions to be executed sequentially;

[0021] The rendering unit is used to render the execution result of the task action after it is executed, and in response to rendering the execution result of the last task action, obtains the execution result of the task instruction.

[0022] Optionally, the large language model task processing device further includes:

[0023] The first display unit is used to display a task planning panel, which displays the progress of the execution of multiple instruction actions;

[0024] The second display unit is used to display a tool component call status panel, which displays the status of calling the tool component.

[0025] The rendering unit is specifically used to: display the execution result panel and render the action execution result on the execution result panel.

[0026] Optionally, the injection unit is specifically used for:

[0027] Obtain general tool component package information, which is information about a component package formed by multiple tool components, and the component package is shared by multiple task actions;

[0028] The knowledge injection component injects the environment description component, tool component information, and general tool component package information corresponding to the target interface into the large language model, so that the large language model generates the tool component call instruction and the general tool component package call instruction corresponding to the task action based on the environment information, tool component information, and general tool component package information in the environment description component.

[0029] The execution unit is specifically used to: execute the tool component invocation instruction and the general tool component package invocation instruction to perform the task action.

[0030] Optionally, the large language model task processing device further includes:

[0031] The allocation unit is used to allocate a component package identifier to the general tool component package;

[0032] The third generation unit is used to generate environmental data for the general tool component package based on the environmental information of each of the multiple tool components.

[0033] The fourth acquisition unit is used to acquire the action type executed by the general tool component package;

[0034] The fourth generation unit is used to generate the general tool component package based on the component package identifier, the environment data, multiple tool components, and the action type.

[0035] Optionally, the injection unit is further configured to:

[0036] Based on the task action and the tool component information, determine the first degree of association between the task action and each tool component in the target interface;

[0037] The tool component information is filtered based on the first correlation to obtain filtered tool component information.

[0038] Based on the task action and the environment description component, a second degree of correlation between the task action and each piece of environmental information is determined;

[0039] The environmental information is filtered based on the second correlation to obtain filtered environmental information.

[0040] The knowledge injection component injects the filtered tool component information and the filtered environment information into the large language model, enabling the large language model to generate the tool component invocation instruction corresponding to the task action based on the filtered tool component information and the filtered environment information.

[0041] Optionally, the injection unit is further configured to:

[0042] The task action is input into the background augmentation model to obtain the first background description of the task action;

[0043] Extract multiple first keywords from the first background description;

[0044] From the tool component information, obtain the first core keyword of each tool component;

[0045] For each tool component, obtain the first distance between each first keyword and the first core keyword of the tool component in the knowledge graph;

[0046] Based on the first distance corresponding to each of the first keywords, the first correlation degree between the task action and the tool component is determined.

[0047] Optionally, the injection unit is further configured to:

[0048] Input the multiple first keywords into the first attention model to obtain the first attention score for each first keyword;

[0049] Based on the first distance and the first attention score corresponding to each of the first keywords, the first correlation between the task action and the tool component is determined.

[0050] Optionally, the injection unit is further configured to:

[0051] Based on the first distance corresponding to the first keyword, a first sub-association degree between the first keyword and the tool component is determined, wherein the first sub-association degree is a decreasing function of the first distance;

[0052] The first attention score corresponding to each first keyword is used to perform a weighted sum operation on the first sub-association degree corresponding to each first keyword to obtain the first association degree between the task action and the tool component.

[0053] Optionally, the injection unit is further configured to:

[0054] Obtain the task action type of the task action;

[0055] Obtain the task plan from which the task action originates, the task plan being generated based on task instructions and including multiple task actions to be executed sequentially;

[0056] The task plan is input into the second attention model to obtain the second attention score of the task action;

[0057] Based on the task action type and the second attention score, a first threshold is determined;

[0058] Based on the comparison between the first correlation degree and the first threshold, the tool component information is filtered to obtain the filtered tool component information.

[0059] Optionally, the injection unit is further configured to:

[0060] The task action is input into the background augmentation model to obtain the first background description of the task action;

[0061] Extract multiple first keywords from the first background description;

[0062] The second core keywords for each environmental information are obtained from the environmental description component.

[0063] For each piece of environmental information, obtain the second distance between each first keyword and the second core keyword of the environmental information in the knowledge graph;

[0064] Based on the second distance corresponding to each of the first keywords, the second correlation degree between the task action and the environmental information is determined.

[0065] Optionally, the injection unit is further configured to:

[0066] Input the multiple first keywords into the first attention model to obtain the first attention score for each first keyword;

[0067] Based on the second distance and the first attention score corresponding to each of the first keywords, the second correlation between the task action and the environmental information is determined.

[0068] Optionally, the injection unit is further configured to:

[0069] Based on the second distance corresponding to the first keyword, a second sub-association degree between the first keyword and the environmental information is determined, wherein the second sub-association degree is a decreasing function of the second distance;

[0070] The second sub-association degree corresponding to each of the first keywords is weighted and summed based on the first attention score corresponding to each of the first keywords to obtain the second association degree between the task action and the environmental information.

[0071] Optionally, the injection unit is further configured to:

[0072] Obtain the task action type of the task action;

[0073] Obtain the task plan from which the task action originates, the task plan being generated based on task instructions and including multiple task actions to be executed sequentially;

[0074] The task plan is input into the second attention model to obtain the second attention score of the task action;

[0075] Based on the task action type and the second attention score, a first threshold is determined;

[0076] Based on the comparison between the second correlation degree and the first threshold, the environmental information is filtered to obtain the filtered environmental information.

[0077] According to one aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the large language model task processing method as described above.

[0078] According to one aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the large language model task processing method as described above.

[0079] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the large language model task processing method as described above.

[0080] In this embodiment, the environment description component and tool component information corresponding to each interface are pre-stored in a storage hierarchy. The tool component information includes basic information about each tool component in the interface. The large language model generates tool component invocation instructions based on task actions, which requires a sufficient understanding of the basic information of each tool component in the interface. The environment description component contains environment information when invoking each tool component in the interface. It describes the environment in which the tool component is invoked and specifies how the results of the tool component invocation are integrated. It is also essential information for the large language model to generate tool component invocation instructions based on task actions. In this embodiment, this information is pre-stored in a storage hierarchy. In response to the task action of the target interface, the environment description component and tool component information corresponding to the target interface are retrieved from the storage hierarchy. Then, the environment description component and tool component information are injected into the large language model through a knowledge injection component. In this way, the large language model can generate tool component invocation instructions based on the environment information in the environment description component and the tool component information, thereby executing the task action. Compared to methods that sift through large amounts of meaningless HTML / CSS / JS text to find key characters, this approach addresses the issue of decreased accuracy in handling excessively long context lengths in large language models. This is because the context description and tool component information corresponding to the target interface is limited, thus improving the processing accuracy of large language models and significantly alleviating the problem of limited context length in related technologies. Furthermore, this method eliminates the need to actually click on each tool component, improving task processing efficiency. Moreover, by avoiding interface cleansing, it enhances the accuracy of large language model task processing.

[0081] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0082] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0083] Figure 1 This is an architecture diagram of a large language model task processing method according to embodiments of the present disclosure;

[0084] Figures 2A-2C This is a schematic diagram illustrating the application of the large language model task processing method according to the embodiments of this disclosure in a shopping platform interface processing scenario;

[0085] Figures 3A-3B This is a schematic diagram illustrating the application of the large language model task processing method according to the embodiments of this disclosure in the interface processing scenario of a reading application.

[0086] Figure 4 This is a general flowchart of a large language model task processing method according to an embodiment of the present disclosure;

[0087] Figures 5A-5B This is a schematic diagram of the storage structure corresponding to the product backend management interface according to an embodiment of this disclosure;

[0088] Figure 6 This is another schematic diagram illustrating the application of the large language model task processing method according to the embodiments of this disclosure in the interface processing scenario of a reading application;

[0089] Figures 7A-7C This is a schematic diagram of general tool component package information according to embodiments of the present disclosure;

[0090] Figures 8A-8B This is a schematic diagram illustrating the determination of task instructions and the execution result of rendering actions according to an embodiment of this disclosure;

[0091] Figure 9 This is a schematic diagram of task planning according to an embodiment of the present disclosure;

[0092] Figures 10A-10D This is a schematic diagram of multiple panels during the execution of task instructions according to an embodiment of the present disclosure;

[0093] Figure 11 This is a schematic diagram of an edge-side intelligent enhancement system that applies a large language model task processing method according to an embodiment of the present disclosure;

[0094] Figure 12 This is a schematic diagram of execution task instructions according to an embodiment of the present disclosure;

[0095] Figure 13This is a block diagram of a large language model task processing apparatus according to embodiments of the present disclosure;

[0096] Figure 14 This is a terminal structure diagram of a method for executing a large language model task processing method according to an embodiment of the present disclosure;

[0097] Figure 15 This is a server architecture diagram of a method for executing large language model task processing according to an embodiment of the present disclosure. Detailed Implementation

[0098] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0099] Large Language Model (LLM) is a large-scale language processing model built on deep learning technology. It has powerful language understanding, generation and reasoning capabilities and is a current research and application hotspot in the field of artificial intelligence.

[0100] Computer use refers to enabling large language models to control the entire computer by manipulating the computer desktop environment, and then interacting with various tools to complete user tasks.

[0101] Intelligent Agent: An autonomous decision-making system based on a large language model, capable of perceiving the environment, planning tasks, and calling tools to perform complex operations.

[0102] Currently, while large language models continue to make breakthroughs and can be generalized to various domains, there has been no substantial progress in tasks requiring user interface manipulation. This is because web interfaces consist of a large amount of HTML, CSS, JavaScript, text, images, and videos. Users only see elements like text, images, and videos, but the HTML, CSS, and JavaScript actually occupy the vast majority of the page's characters. Manipulating such interfaces requires finding the necessary key characters within a large amount of meaningless text. Large language models can only handle limited context length, and due to the limitations of attention mechanisms, excessively long contexts can lead to partial memory loss, resulting in serious problems such as decreased response quality.

[0103] To address this issue, one related technique cleanses the interface data using methods such as regular expression matching. While this reduces the context length, it can lead to the loss of crucial information, reducing the accuracy of large language model processing tasks. Another related technique acquires tool component information and background information from the interface by simulating element clicks. For example, tool components are identified within the interface's HTML / CSS / JS / text / images / videos. By actually clicking on these tool components, information about the component and its execution environment is obtained, providing accurate tool component and environment information for large language model processing tasks. However, actually clicking on each tool component results in low task processing efficiency.

[0104] Based on this, embodiments of this disclosure provide a method, apparatus, electronic device, and medium for processing large language model tasks. The large language model task processing method provided by embodiments of this disclosure can improve the accuracy of task processing while maintaining the efficiency of large language model task processing.

[0105] System architecture and scenario description of the embodiments disclosed herein

[0106] Figure 1 This is a system architecture diagram of the large language model task processing method according to the embodiments of this disclosure. Figure 1 This includes terminal 140, Internet 130, gateway 120, server 110, etc.

[0107] Terminal 140 is a device that displays the target interface. Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0108] Server 110 refers to a computer system capable of providing large language model task processing services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). In a distributed system, multiple servers 110 are interconnected as computing and storage units to transmit data and coordinate task processing using shared communication lines. Server 140 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0109] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0110] The embodiments disclosed herein can be applied in various scenarios, such as with Figures 2A-2C The shopping platform interface shown Figures 3A-3B The examples shown depict scenarios such as large language model task processing related to the reading application interface.

[0111] Figure 2A The shopping platform interface shown is composed of a large amount of HTML (Hypertext Markup Language), CSS (Cascading Style Sheets), JS (Interaction Logic), text, images, and videos. Users can only see elements such as text, images, and videos, but in reality, the HTML, CSS, and JS of the page occupy the vast majority of the characters. Figure 2B yes Figure 2A The code page corresponding to the shopping platform interface shown is from... Figure 2B This indicates that the total character length of the shopping platform's interface is approximately 639,000 characters. Additionally, Figure 2A The character corresponding to product 2 is Figure 2B The text describes a problem involving multiple characters within a rectangular frame. It states that there are only a few hundred characters corresponding to product 2, making finding the key text from 639,000 characters extremely difficult, and that the maximum context length of large language models in related technologies is only 128,000 characters. Figure 2C The storage hierarchy in the memory stores Figure 2A The shopping platform interface shown includes information about the environment description components and tool components. For example, the product search component information and the environment description component "search box, used to receive search text" are shown. Figure 2A The search box corresponds to this. The task action is for... Figure 2A The clicking function for product 2 on the shopping platform interface will therefore... Figure 2C The environment description component and tool component information in the storage hierarchy shown are injected into the large language model. Based on this, the large language model can generate the calling instruction of the product click component 2 corresponding to the task action "click product 2". Thus, the large language model realizes the task processing of the shopping platform interface.

[0112] Figure 3A This is a screenshot of the interface of a mobile reading application. Figure 3BThe storage hierarchy shown stores Figure 3A The interface shown is an environment description component and tool component information corresponding to a reading application. Assume the task action targets... Figure 3A The interface settings of the reading application shown are as follows, and the specific task action is "Search for book X". Therefore, Figure 3B The environment description component and tool component information in the storage hierarchy shown are injected into the large language model. Based on this, the large language model can generate the call instructions for the search box component and data search component corresponding to the task action "search for book X". Thus, the large language model realizes the task processing of reading application interface.

[0113] It should be understood that the above description only illustrates some application scenarios of this disclosure. The business scenarios to which this disclosure can be applied may include, but are not limited to, the specific embodiments described above.

[0114] General Description of Embodiments in this Disclosure

[0115] It should be emphasized that the embodiments disclosed herein are applicable to a variety of application scenarios, such as with Figures 2A-2C The large language model task processing and... (The sentence is incomplete and requires more context to translate accurately.) Figures 3A-3B The above describes large language model task processing related to a reading application interface. In related technologies, large language models can only handle limited context lengths. Furthermore, due to limitations in attention mechanisms, excessively long contexts can lead to partial memory loss, resulting in serious problems such as decreased answer quality. One related technology cleanses the interface data using methods such as regular expression matching, reducing the context length after cleaning. However, cleaning may result in the loss of key information, reducing the accuracy of large language model task processing. Another related technology obtains tool component information and background information in the interface by simulating element clicks. By actually clicking on the tool component, information about that tool component and the execution environment is obtained, thus providing accurate tool component and environment information for large language model task processing. However, actually clicking on each tool component leads to low task processing efficiency. Some embodiments of this disclosure provide a large language model task processing method, apparatus, electronic device, and medium that can improve task processing accuracy while maintaining the efficiency of large language model task processing.

[0116] The large language model task processing method responds to task actions on the target interface by generating tool components from the large language model to invoke instructions and execute task actions. This method can improve the accuracy of task processing while maintaining the efficiency of large language model task processing.

[0117] The large language model task processing method of this disclosure can be performed on the terminal 140, on the server 110, or partially on the terminal 140 and partially on the server 110. Furthermore, the large language model task processing method of this disclosure is applicable not only to task processing on various web interfaces but also to task processing on in-application interfaces.

[0118] like Figure 4 As shown, a large language model task processing method according to an embodiment of this disclosure may include:

[0119] Step 410: In response to the task action on the target interface, retrieve the environment description component and tool component information of the target interface from the storage hierarchy. The storage hierarchy stores the environment description component and tool component information according to the interface. The environment description component contains the environment information when calling each tool component in the interface.

[0120] Step 420: Through the knowledge injection component, inject the environment description component and tool component information corresponding to the target interface into the large language model, so that the large language model can generate the tool component call instructions corresponding to the task action based on the environment information in the environment description component and the tool component information.

[0121] Step 430: Execute the tool component invocation command to perform the task action.

[0122] Steps 410-430 are described in detail below.

[0123] In step 410, in response to the task action on the target interface, the environment description component corresponding to the target interface and the tool component information of the target interface are obtained from the storage hierarchy. The storage hierarchy stores the environment description component and tool component information according to the interface. The environment description component contains the environment information when calling each tool component in the interface.

[0124] The target interface is the interface corresponding to the large language model task. The large language model task requires manipulation of the target interface for processing. (See reference...) Figures 2A-2C The target interface refers to Figure 2A The shopping platform interface shown. (Refer to...) Figures 3A-3B The target interface refers to Figure 3A The interface shown is for a reading application. Figure 5A The product backend management interface shown can also be used as a target interface.

[0125] Task actions are actions performed on the target interface, and these actions are set as needed. Task actions can be actions directly entered by the user, or actions determined based on parsed text input from the user. For example, for... Figure 3AThe target interface shown assumes that if the user directly enters "Search for book X", then "Search for book X" can be directly used as the task action. For... Figure 2A If the user enters "summarize the prices and parameters of multiple existing mobile phone brands" on the target interface shown, then the text entered by the user needs to be parsed to determine the task action, which includes "click on product 2" and "click on product 1".

[0126] The environment description component contains environment information when calling various tool components in the interface. It describes the environment in which the tool components are called and specifies how to integrate the results of the tool components after they are called. Figure 5B The environment description components shown are Figure 5A Correspondingly, the environment description component includes components a, b, and c. Component a corresponds to the product search component, describing how the product search component queries orders located between the time range startTime and endTime. Component a specifies how the query results are integrated into an order list based on the results of the invoked tool component. Component b corresponds to the product list component, describing how, based on the order list, the cost price and selling price of each product can be determined from the product list, and a product details list can be generated. Component c corresponds to the profit calculation component, specifying that the profit calculation component can calculate the total profit of each product included in the product details list. Additionally, the environment description component may also include a comprehensive description of the target interface, i.e., the global context of the target interface, for example... Figure 5B The environment description component shown also describes Figure 5A The interface shown is the order management interface of the xx shopping platform, which includes a product search component, a product list component, and a profit calculation component.

[0127] The tool component information contains basic information about each tool component in the target interface. Figure 5A The tool component information corresponding to the product backend management interface shown is as follows: Figure 5B As shown, the three tool components are related to the product search component, product list component, and profit calculation component, respectively. Based on the tool component information, it can be determined that the product search component includes the order query interface `queryOrderList`, which accepts the `startTime` and `endTime` parameters. The product list component queries the product details interface `goodInfo` for parameters such as cost price and selling price, and the `goodInfo` interface accepts the `goodId` (product identifier) ​​parameter. The profit calculation component includes the profit calculation interface `profitCalculation`, which accepts the `costPrice` and `sellingPrice` parameters.

[0128] A storage hierarchy refers to a structure that stores information about at least one environment description component and tool component corresponding to an interface, and the environment description component and tool component information are stored in the storage hierarchy according to the interface. In the large language model task processing method of this disclosure embodiment, the storage hierarchy structure at least includes information about the environment description component and tool component corresponding to the target interface. In one embodiment, Figure 5A The product backend management page shown is the same as Figure 2A The shopping platform interfaces shown belong to the same shopping platform, therefore... Figure 2C and Figure 5B The storage hierarchy shown is part of the storage hierarchy corresponding to the shopping platform.

[0129] If it's determined that the task action is executed on a target interface, the environment description component and tool component information corresponding to the target interface can be retrieved from the storage hierarchy. For example, for the XX shopping platform, if the task action targets... Figure 2A When the interface shown is executed, it retrieves data from the storage hierarchy. Figure 2C The environment description components and tool component information are shown. If the task action targets... Figure 5A When the interface shown is executed, it retrieves data from the storage hierarchy. Figure 5B The environment description components and tool component information are shown.

[0130] In step 420, the knowledge injection component injects the environment description component and tool component information corresponding to the target interface into the large language model, so that the large language model generates the tool component call instructions corresponding to the task action based on the environment information in the environment description component and the tool component information.

[0131] The knowledge injection component is a component that injects environment description component and tool component information into the large language model. The knowledge injection component can centrally manage and schedule the environment description component and tool component information corresponding to each interface stored in the storage hierarchy, and synchronize the environment description component and tool component information corresponding to the target node to the large language model.

[0132] Tool component invocation instructions are instructions that invoke the tool components corresponding to the task actions.

[0133] The large language model analyzes and identifies environmental information and tool component information in the environment description component to determine the tool component corresponding to the task and generates the corresponding tool component invocation instructions. (See reference...) Figure 2C The large language model analyzes and identifies the environmental information and tool component information in the environment description component, determining that the tool component corresponding to the task action "click product 2" is product click component 2, and generates the tool component invocation instruction corresponding to product click component 2. (See reference...) Figure 3B The large language model analyzes and identifies the environmental information and tool component information in the environment description component to obtain the search box component and book search component corresponding to the task action "search for book X", and generates the two corresponding tool component call instructions. The execution order of the tool component call instructions corresponding to the search box component is before that of the data search component.

[0134] In step 430, the tool component invocation instruction is executed to perform the task action.

[0135] After determining the tool component invocation instruction corresponding to the task action, the tool component invocation instruction is executed to perform the task action.

[0136] Reference Figure 2C After generating the tool component invocation instructions for product click component 2 using the large language model, the task action is executed, namely, clicking the control corresponding to product 2 to display the product 2 details page. (Refer to...) Figure 3B After generating the tool component call command for searching book X through the large language model, the task action is executed, namely, searching for book X and displaying the search results for book X.

[0137] It should be noted that large language models in related technologies suffer from the illusion problem, which is particularly severe on mobile devices such as smartphones, where memory limitations generally limit the execution of smaller models. The embodiments disclosed in this paper can inject accurate data into the large language model, namely information from the environment description component and tool component, which can greatly alleviate the illusion problem, improve its usability, and enhance the processing accuracy of large language model tasks.

[0138] In the embodiments described in steps 410-430 above, the environment description component and tool component information corresponding to each interface are pre-stored in the storage hierarchy. The tool component information includes basic information about each tool component in the interface. The large language model generates tool component invocation instructions based on task actions, which requires a sufficient understanding of the basic information of each tool component in the interface. The environment description component contains environment information when calling each tool component in the interface. It describes the environment in which the tool component is called and specifies how to integrate the results of the tool component call. It is also the information that the large language model must rely on to generate tool component invocation instructions based on task actions. In this embodiment, this information is pre-stored in the storage hierarchy. In response to the task action of the target interface, the environment description component and tool component information corresponding to the target interface are obtained from the storage hierarchy. Then, the environment description component and tool component information are injected into the large language model through the knowledge injection component. In this way, the large language model can generate tool component invocation instructions based on the environment information in the environment description component and the tool component information, thereby executing the task action. Compared to methods that sift through large amounts of meaningless HTML / CSS / JS text to find key characters, this approach addresses the issue of decreased accuracy in handling excessively long context lengths in large language models. This is because the context description and tool component information corresponding to the target interface is limited, thus improving the processing accuracy of large language models and significantly alleviating the problem of limited context length in related technologies. Furthermore, this method eliminates the need to actually click on each tool component, improving task processing efficiency. Moreover, by avoiding interface cleansing, it enhances the accuracy of large language model task processing.

[0139] The above is a general description of steps 410-430. Since step 410 has been described in sufficient detail above, the specific implementation process of steps 420 and 430 will be described in detail below.

[0140] Detailed description of step 420

[0141] In step 420, the knowledge injection component injects the environment description component and tool component information corresponding to the target interface into the large language model, so that the large language model generates the tool component call instructions corresponding to the task action based on the environment information in the environment description component and the tool component information.

[0142] In one embodiment, step 420 includes:

[0143] Step 510: Based on the task action and tool component information, determine the first degree of association between the task action and each tool component in the target interface;

[0144] Step 520: Filter the tool component information based on the first relevance to obtain the filtered tool component information;

[0145] Step 530: Based on the task action and environment description components, determine the second degree of correlation between the task action and each piece of environmental information;

[0146] Step 540: Filter the environmental information based on the second correlation degree to obtain the filtered environmental information;

[0147] Step 550: Through the knowledge injection component, the filtered tool component information and the filtered environment information are injected into the large language model, so that the large language model can generate tool component call instructions corresponding to the task actions based on the filtered tool component information and the filtered environment information.

[0148] Steps 510-550 are described in detail below.

[0149] In step 510, based on the task action and tool component information, the first degree of association between the task action and each tool component in the target interface is determined.

[0150] The first degree of correlation corresponds to the tool components. The first degree of correlation refers to the degree of correlation between the task action and each tool component in the target interface. The first degree of correlation is determined based on the task action and the tool component information corresponding to each tool component in the target interface.

[0151] Reference Figure 2C The tool component information includes product search component information, product click component 2 information, etc. Based on the product search component information and task actions, the first degree of association between the product search component and "click product 2" can be determined. Based on the product click component 2 information and task actions, the first degree of association between the product click component 2 and "click product 2" can be determined.

[0152] The primary correlation can also be considered as the matching degree between task actions and various tool components. The higher the primary correlation, the greater the likelihood that a tool component is the corresponding tool component for the task action. Figure 2C In the data, the first correlation between the product click component 2 and the task action "click component" is significantly greater than the first correlation between the product search component and the task action.

[0153] In step 520, the tool component information is filtered based on the first correlation degree to obtain the filtered tool component information.

[0154] The filtered tool component information is at least one tool component obtained by filtering tool component information based on the first relevance. The tool component corresponding to the filtered tool component information has a significantly higher first relevance to the task action than other tool components.

[0155] The number of filtered tool component information can be set as needed. Tool component information corresponding to one or more tool components with the highest first relevance can be used as filtered tool component information.

[0156] In step 530, based on the task action and environment description component, a second degree of correlation between the task action and each piece of environmental information is determined.

[0157] The second correlation degree corresponds to each piece of environmental information in the environment description component. It refers to the degree of association between the task action and each piece of environmental information. The environment component information includes the environment information when calling each tool component in the target interface. Therefore, the second correlation degree can represent the degree of association between the task action and the tool component corresponding to the environment information. The higher the second correlation degree between the task action and the environment information, the greater the likelihood that the tool component corresponding to the environment information is the tool component corresponding to the task action.

[0158] Reference Figure 2C The environment description component includes a search box (for receiving search text) and environment description component N. Based on the specific environment information of the search box, its second degree of association with the task action "click on product 2" can be determined. Based on the specific description corresponding to environment description component N, the second degree of association between the environment information corresponding to environment description component N and the task action "click on product 2" can be determined.

[0159] In step 540, the environmental information is filtered based on the second correlation degree to obtain filtered environmental information.

[0160] The filtered environmental information is at least one piece of environmental information obtained by filtering environmental information based on the second correlation. The second correlation between the filtered environmental information and the task action is significantly higher than that of other environmental information.

[0161] The number of filtered environmental information entries can be set as needed. One or more environmental information entries with the highest second correlation can be used as filtered environmental information entries.

[0162] In step 550, filtered tool component information and filtered environment information are injected into the large language model through the knowledge injection component, so that the large language model can generate tool component invocation instructions corresponding to the task action based on the filtered tool component information and filtered environment information.

[0163] The tool component information corresponding to the target interface is initially filtered using a first degree of relevance, resulting in filtered tool component information. Then, multiple environment information items in the environment description component corresponding to the target interface are filtered using a second degree of relevance, resulting in filtered environment information. Next, the filtered tool component information and filtered environment information are injected into the large language model via a knowledge injection component. The large language model can then determine the tool components corresponding to the task based on these two sets of information and generate the corresponding tool component invocation information.

[0164] Assume the target interface's tool component information includes tool component information a, tool component information b, tool component information c, and tool component information d, while the filtered tool component information includes tool component information b and tool component information c. Additionally, the target interface's environment description component includes environment information a, environment information b, environment information c, and environment information d, while the filtered environment information includes environment information c and environment information d. Then, we only need to inject tool component information b, tool component information c, and environment information c and d into the large language model.

[0165] The embodiments of steps 510-550 above filter tool component information based on the first correlation between task actions and each tool component, obtaining filtered tool component information. The first correlation between the filtered tool component information and the task action is higher than that of other tool components. Furthermore, based on the second correlation between task actions and various environmental information, multiple environmental information items in the environment description component are filtered, obtaining filtered environment information. The second correlation between the filtered environment information and the task action is significantly higher than that of other environment information. These two filtering processes effectively reduce the amount of tool component information and environment information that the large language model needs to process, further improving the processing efficiency of interface tasks. Moreover, the higher correlation between the filtered tool component information and the filtered environment information and the task action further improves the accuracy of the large language model's task processing.

[0166] The above is a general description of steps 510-550. Since step 550 has been described in sufficient detail above, the specific implementation process of steps 510-540 will be described in detail below.

[0167] In step 510, based on the task action and tool component information, the first degree of association between the task action and each tool component in the target interface is determined.

[0168] In one embodiment, step 510 includes:

[0169] Step 610: Input the task action into the background augmentation model to obtain the first background description of the task action;

[0170] Step 620: Extract multiple primary keywords from the primary background description;

[0171] Step 630: Obtain the first core keyword for each tool component from the tool component information;

[0172] Step 640: For each tool component, obtain the first distance between each first keyword and the first core keyword of the tool component in the knowledge graph;

[0173] Step 650: Based on the first distance corresponding to each first keyword, determine the first degree of association between the task action and the tool component.

[0174] Steps 610-650 are described in detail below.

[0175] In step 610, the task action is input into the background augmentation model to obtain the first background description of the task action.

[0176] The background augmentation model is a model that adds background information to a task action. The first background description is the augmented background of the task action. The input of the background augmentation model is the task action, and the output of the background augmentation model is the first background description. Compared with the task action, the first background description contains more and richer effective information.

[0177] Reference Figure 2C For the task action "click on product 2", this task action is input into the background augmentation model, which can be based on... Figure 2A The interface shown expands the task action to obtain the first background description of the task: "Click on the XX shopping platform to search for the text 'mobile phone' and find product 2 on the corresponding search page. The full name of product 2 is...".

[0178] Reference Figure 3B For the task action "search for book X", a background augmentation model is used to augment it, and the background augmentation model is based on... Figure 3A The interface shown is expanded with a background, resulting in the first background description: "Search for book X in the bookstore".

[0179] In step 620, multiple first keywords are extracted from the first background description.

[0180] The first keyword is a keyword extracted from the first background description. The first keyword contains information about the task action. Suppose the first background description is "Search for book X in the bookstore", the first keyword could be "bookstore", "search", or "book X".

[0181] In step 630, the first core keyword of each tool component is obtained from the tool component information.

[0182] The first core keyword corresponds to the tool component and can be used to describe the main function of the tool component. For example, if the tool component information is "Product Search Component: Includes queryOrderList (order query interface, accepting startTime and endTime parameters)," then the first core keyword could be order query. Similarly, if the tool component information is "Search Component Information Includes Book Search Interface," then the first core keyword could be book search.

[0183] In step 640, for each tool component, the first distance between each first keyword and the first core keyword of the tool component in the knowledge graph is obtained.

[0184] The first distance refers to the distance between the primary keywords corresponding to task actions and the primary core keywords of tool components in the knowledge graph. The first distance can be used to measure the semantic correlation between the primary keywords and the primary core keywords, thereby determining the degree of association between the primary keywords and the primary core keywords. The smaller the first distance, the higher the degree of association between the primary keywords and the primary core keywords.

[0185] A knowledge graph is a structured semantic network used to represent and organize knowledge. It can describe the relationship between the primary keyword and the primary core keyword through entities, relations, and attributes.

[0186] The first distance can be calculated using methods such as shortest path, semantic similarity, and graph feature measurement. Assuming the first keyword and the first core keyword are not directly connected in the knowledge graph, but are similar through context or attributes, the distance can be measured using semantic similarity. First, the first keyword and the first core keyword are mapped to low-dimensional vectors, and then cosine similarity or Euclidean distance is used to determine the first distance.

[0187] In step 650, the first correlation between the task action and the tool component is determined based on the first distance corresponding to each first keyword.

[0188] Each task action corresponds to multiple primary keywords. Therefore, to determine the primary correlation between a task action and a tool component, it is necessary to determine the primary distance between each primary keyword and the primary keyword of the tool component.

[0189] The first degree of association can be determined in various ways, and it is inversely proportional to the first distance. In one embodiment, the difference between 1 and each of the first distances is calculated, and the average or weighted sum of the multiple differences is taken as the first degree of association. Alternatively, the first degree of association can be determined based on lookup tables, mapping functions, etc.

[0190] Reference Figure 3A and Figure 3BGiven the task action "search for book X", the background expansion model can be used to expand the task action, thereby identifying the three primary keywords corresponding to the task action: "bookstore", "search", and "book X". Then, the first distance between these three primary keywords and the primary core keywords corresponding to the tool components on the knowledge graph is calculated, thus determining the first correlation between the task action and the tool components. Since the primary core keyword in both the search box component information and the book search component information contains "search", based on the first correlation, the filtered component information can be determined to be the search box component information and the book search component information.

[0191] Reference Figure 3A and Figure 6 Given the task action "search for book E", the background expansion model can expand the task action to identify three primary keywords: "bookshelf", "search", and "book E". Then, the first distance between these three primary keywords and the primary core keywords corresponding to the tool components on the knowledge graph is calculated to determine the first correlation between the task action and the tool components. The primary core keyword in both the search box component information and the search book component information contains "search", while the book E reading component information contains "read". Therefore, based on the first correlation, the filtered component information can be the search box component information, the search book component information, and the book E reading component information. Finally, the large language model determines the correspondence between the task action "search for book X" and the book E reading component information based on the filtered component information and the filtered environment information, thereby generating the tool component invocation command for the book E reading component. Figure 3A , Figure 3B and Figure 6 It can be determined that even when task actions are similar, the final tool component corresponding to the task action may be different, thus enabling the generation of tool component invocation instructions that are more closely matched to the task action.

[0192] The embodiments of steps 610-650 above first expand the background of the task action to obtain a first background description, and then extract multiple first keywords from the first background description. Compared with the task action, the first background description obtained through background expansion effectively increases the amount of information, thereby increasing the amount of information contained in the multiple first keywords. Next, the first core keywords of each tool component are obtained from the tool component information. Based on the first core keywords, the main functions of the tool component can be determined. For each tool component, the first distance between each first keyword and the first core keyword of the tool component in the knowledge graph is obtained to represent the semantic association degree between the first keyword and the first core keyword. Based on the first distance corresponding to each first keyword, the first association degree between the task action and the tool component can be determined. The setting of multiple first keywords effectively increases the amount of information, thereby improving the accuracy of the first distance, improving the accuracy of the first association degree, improving the accuracy of the filtered tool component information, and improving the efficiency and accuracy of large language model task processing.

[0193] In step 650, the first correlation between the task action and the tool component is determined based on the first distance corresponding to each first keyword.

[0194] In one embodiment, step 650 includes:

[0195] Step 710: Input multiple primary keywords into the primary attention model to obtain the primary attention score for each primary keyword;

[0196] Step 720: Based on the first distance and first attention score corresponding to each first keyword, determine the first correlation between the task action and the tool component.

[0197] Steps 710-720 are described in detail below.

[0198] In step 710, multiple first keywords are input into the first attention model to obtain the first attention score for each first keyword.

[0199] The first attention score corresponds to the first keyword. It's a crucial metric in the first attention model for measuring the relevance between different first keywords, quantifying their importance. A higher first attention score indicates greater importance of the first keyword, leading to increased attention paid to it during subsequent UI task processing.

[0200] The first attention model is a model that determines the first attention score for each first keyword. The first attention model can be a transformer model, a pre-trained language model (Bidirectional Encoder Representations from Transformers, BERT), etc.

[0201] Suppose that the primary keywords corresponding to the task action "search for book X" are "bookstore", "search", and "book X", inputting the primary keywords into the primary attention model can yield primary attention scores corresponding to "bookstore", "search", and "book X" respectively.

[0202] In step 720, the first correlation between the task action and the tool component is determined based on the first distance and the first attention score corresponding to each first keyword.

[0203] The initial correlation between task actions and tool components is determined based on the initial distance and initial attention score corresponding to each initial keyword.

[0204] There are various ways to determine the first degree of relevance, and it is inversely proportional to the first distance and directly proportional to the first attention score. The first degree of relevance can be determined using methods such as table lookup and function mapping. When determining the first degree of relevance using table lookup, the range of values ​​for the first distance and the first attention score can be divided into multiple intervals. Each interval has a corresponding first distance, first attention score, and first degree of relevance. Therefore, based on the interval containing the first attention score and the interval containing the first distance, the first degree of relevance between the task action and the tool component can be directly determined.

[0205] In the embodiments of steps 710-720 above, the first attention score is taken into account in the process of determining the first relevance based on the first distance corresponding to each first keyword. The first attention score is the processing result of the attention model on multiple first keywords. The first attention score can measure the importance of each first keyword. Therefore, the accuracy of the first relevance between the task action and the tool component determined based on the first distance and the first attention score corresponding to the first keyword is higher, thereby improving the accuracy of the large language model task processing.

[0206] In one embodiment, step 720 includes:

[0207] Step 810: Based on the first distance corresponding to the first keyword, determine the first sub-association degree between the first keyword and the tool component. The first sub-association degree is a decreasing function of the first distance.

[0208] Step 820: Using the first attention score corresponding to each first keyword, perform a weighted sum operation on the first sub-association degree corresponding to each first keyword to obtain the first association degree between the task action and the tool component.

[0209] Steps 810-820 are described in detail below.

[0210] In step 810, based on the first distance corresponding to the first keyword, the first sub-association degree between the first keyword and the tool component is determined. The first sub-association degree is a decreasing function of the first distance.

[0211] The first sub-association degree is the degree of association between the first keyword and the tool component, and it is determined based on the first distance. Furthermore, the degree of association between the first keyword and the tool component is inversely proportional to the first distance; therefore, the first sub-association degree is a decreasing function of the first distance.

[0212] The function between the first sub-association degree and the first distance can be set arbitrarily, as long as the first sub-association degree is a decreasing function of the first distance. This function can be y = 1 - x, y = 1 / x, etc., where y is the first sub-association degree and x is the first distance.

[0213] In step 820, the first sub-associations corresponding to each first keyword are weighted and summed based on the first attention score corresponding to each first keyword to obtain the first association between the task action and the tool component.

[0214] The first attention score represents the importance of the first associated word among multiple first keywords. Therefore, the first attention score can be used as the weight of the first keyword to perform a weighted sum operation on the first sub-associations corresponding to each first keyword, thereby determining the first association between the task action and the tool component.

[0215] It should be noted that the sum of the first attention scores corresponding to each first keyword is 1. If the sum of the first attention scores of each first keyword output by the attention model is not 1, the first attention scores can be transformed to meet the requirement that the sum is 1.

[0216] Suppose that the first keywords corresponding to the task action "search for book X" are "bookstore", "search", and "book X", where the first attention score of the first keyword "bookstore" is 0.3 and the first sub-association is 0.8, the first attention score of the first keyword "search" is 0.3 and the first sub-association is 0.5, and the first attention score of the first keyword "book X" is 0.4 and the first sub-association is 0.9, then the first association between the task "search for book X" and the tool component is 0.75.

[0217] The embodiments of steps 810-820 above determine the first sub-association degree between the first keyword and the tool component based on the first distance corresponding to the first keyword. Then, the first sub-association degree corresponding to each first keyword is weighted and summed using the first attention score corresponding to each first keyword to obtain the first association degree between the task action and the tool component. Since the first attention score can reflect the importance of the first keyword among multiple first keywords, using the first attention score as the weight in the weighted summation process can improve the accuracy of the result corresponding to the weighted summation, i.e., the first association degree, thereby improving the accuracy of the filtered tool component information and improving the accuracy of the large language model task processing.

[0218] In one embodiment, step 520 includes:

[0219] Step 910: Obtain the task action type of the task action;

[0220] Step 920: Obtain the task plan from which the task action originates. The task plan is generated based on the task instructions and includes multiple task actions to be executed sequentially.

[0221] Step 930: Input the task plan into the second attention model to obtain the second attention score of the task action;

[0222] Step 940: Determine the first threshold based on the task action type and the second attention score;

[0223] Step 950: Based on the comparison between the first correlation degree and the first threshold, filter the tool component information to obtain the filtered tool component information.

[0224] Steps 910-950 are described in detail below.

[0225] In step 910, the task action type of the task action is obtained.

[0226] The task action type is the type of action of the task action. If the task action is "search for book X", then the action type is search. If the task action is "click on product 2", then the action type is product click or product view.

[0227] In step 920, the task plan from which the task action is derived is obtained. The task plan is generated based on the task instructions and includes multiple task actions to be executed sequentially.

[0228] Task instructions are generated based on the text input by the user. For example, if a user inputs "Calculate the profit of goods this week" into the large language model, the task instruction would be to calculate the profit of goods from [Month X, 2025] to [Month X+7, 2025]. Similarly, if a user inputs "Open book X" into the large language model, the task instruction could be "Open book X".

[0229] Task planning is the planning of multiple tasks generated based on task instructions. Based on task planning, multiple tasks to be executed and the execution order of these tasks can be determined. Assuming the task instruction is "Open book X", the task planning includes the sequentially executed task actions "Search book X" and "Open book X".

[0230] In step 930, the task planning is input into the second attention model to obtain the second attention score of the task action.

[0231] The second attention score corresponds to the task action and is an important indicator used by the second attention model to measure the correlation between various task actions, quantifying the importance of each action. A higher second attention score indicates greater importance of the task action, leading to increased attention paid to that action during subsequent interface task processing.

[0232] The second attention model is a model that determines the second attention score of task actions. The second attention model can be a transformer model, a pre-trained language model (Bidirectional Encoder Representations from Transformers, BERT), etc.

[0233] Assuming the task planning includes the sequentially executed task actions "search for book X" and "open book X", the task planning is input into the second attention model to obtain the second attention score for the task action "search for book X" and the second attention score for the task action "open book X".

[0234] In step 940, a first threshold is determined based on the task action type and the second attention score.

[0235] The first threshold is a standard for filtering tool component information, and the first threshold is determined based on the task action type and the second attention score. The first threshold corresponds one-to-one with the task action type.

[0236] In determining the first threshold, the task action type first needs to be converted into a type score. Specifically, the type score corresponding to the task action type can be determined by looking up a table. Then, based on the function corresponding to the type score and the second attention score, the first threshold is determined, and the higher the type score and the second attention score, the higher the first threshold.

[0237] Assume the task planning includes the sequentially executed tasks "Search for book X" and "Open book X". The task action type for "Search for book X" is a search action, and the task action type for "Open book" is a book click action. First, determine the type score for the search action type as 0.6 and the type score for the book click action type as 0.7. Simultaneously, input the task planning into the second attention model, obtaining a second attention score of 0.6 for "Open book X" and 0.4 for "Search for book X". If the first threshold is determined based on the average of the type score and the second attention score, then the first threshold for the task action "Search for book X" is 0.5, and the first threshold for the task action "Open book X" is 0.65.

[0238] It should be noted that the mapping function between the first threshold and the task action type and the second attention score can be set as needed. This mapping function can be the average or product of the type score corresponding to the task action type and the second attention score.

[0239] In step 950, the tool component information is filtered based on the comparison between the first correlation degree and the first threshold to obtain the filtered tool component information.

[0240] After determining the first threshold, for each task action, the first correlation degree corresponding to each tool component is compared with the first threshold to filter the tool component information, and the tool component information with the first correlation degree greater than or equal to the first threshold is used as the filtered component information.

[0241] In the embodiments of steps 910-950 above, the tool component information is filtered based on the comparison between the first threshold and the first correlation degree of each tool component to obtain filtered component information. The first threshold is determined based on the task action type corresponding to the task action and the second attention score. The second attention score is the processing result of the second attention model on the task planning from which the task action comes. Therefore, the second attention score can determine the importance of the task action in the task planning, and the importance of the action type of the task action can be determined based on the task action type. Thus, the first threshold is more matched with the task action, thereby improving the accuracy of the filtered component information and improving the accuracy of the large language model task processing.

[0242] In one embodiment, step 530 includes:

[0243] Step 1010: Input the task action into the background augmentation model to obtain the first background description of the task action;

[0244] Step 1020: Extract multiple primary keywords from the primary background description;

[0245] Step 1030: Obtain the second core keywords for each environmental information from the environmental description component;

[0246] Step 1040: For each piece of environmental information, obtain the second distance between each primary keyword and the second core keyword of the environmental information in the knowledge graph;

[0247] Step 1050: Based on the second distance corresponding to each first keyword, determine the second correlation between task actions and environmental information.

[0248] Steps 1010-1050 are described in detail below.

[0249] In step 1010, the task action is input into the background augmentation model to obtain the first background description of the task action.

[0250] The background augmentation model is a model that adds background information to a task action. The first background description is the augmented background of the task action. The input of the background augmentation model is the task action, and the output of the background augmentation model is the first background description. Compared with the task action, the first background description contains more and richer effective information.

[0251] Reference Figure 2C For the task action "click on product 2", this task action is input into the background augmentation model, which can be based on... Figure 2A The interface shown expands the task action to obtain the first background description of the task: "Click on the XX shopping platform to search for the text 'mobile phone' and find product 2 on the corresponding search page. The full name of product 2 is...".

[0252] Reference Figure 3B For the task action "search for book X", a background augmentation model is used to augment it, and the background augmentation model is based on... Figure 3A The interface shown is expanded with a background, resulting in the first background description: "Search for book X in the bookstore".

[0253] In step 1020, multiple first keywords are extracted from the first background description.

[0254] The first keyword is a keyword extracted from the first background description. The first keyword contains information about the task action. Suppose the first background description is "Search for book X in the bookstore", the first keyword could be "bookstore", "search", or "book X".

[0255] In step 1030, the second core keywords of each environmental information are obtained from the environmental description component.

[0256] The second core keyword corresponds to the environmental information, which can be understood as the main content within that information. For example, if the environmental information is "Based on the order list, determine the cost price and selling price of each product, and generate a product details list," then the second core keyword could be "product details list." If the environmental information is "Calculate the total profit of each product included in the product details list," then the second core keyword could be "profit calculation."

[0257] The second core keyword can be a keyword in environmental information, or a summary of environmental information.

[0258] In step 1040, for each piece of environmental information, the second distance between each first keyword and the second core keyword of the environmental information in the knowledge graph is obtained.

[0259] The second distance is the distance between the primary keywords corresponding to the task actions and the secondary core keywords of the environmental information in the knowledge graph. The second distance can be used to measure the semantic correlation between the primary keywords and the secondary core keywords, thereby determining the degree of association between the primary keywords and the secondary core keywords. The smaller the second distance, the higher the degree of association between the primary keywords and the secondary core keywords.

[0260] A knowledge graph is a structured semantic network used to represent and organize knowledge. It can describe the relationship between the primary keyword and the secondary core keyword through entities, relations, and attributes.

[0261] The second distance can be calculated using methods such as shortest path, semantic similarity, and graph feature metrics. Assuming the first keyword and the second core keyword are not directly connected in the knowledge graph, but are similar through context or attributes, semantic similarity can be used to measure the distance. First, the first keyword and the second core keyword are mapped to low-dimensional vectors respectively, and then cosine similarity or Euclidean distance is used to determine the second distance.

[0262] In step 1050, the second correlation degree between task actions and environmental information is determined based on the second distance corresponding to each first keyword.

[0263] The task action corresponds to multiple primary keywords. Therefore, if it is necessary to determine the secondary correlation between the task action and the environmental information, it is necessary to determine it based on the secondary distance between the primary keywords and the secondary core keywords of the environmental information.

[0264] The second correlation degree can be determined in various ways, and it is inversely proportional to the second distance. In one embodiment, the difference between 1 and each of the second distances is calculated, and the average or weighted sum of the multiple differences is taken as the second correlation degree. Alternatively, the second correlation degree can be determined based on lookup tables, mapping functions, etc.

[0265] The embodiments of steps 1010-1050 above first expand the background of the task action to obtain a first background description, and then extract multiple first keywords from the first background description. Compared with the task action, the first background description obtained through background expansion effectively increases the amount of information, thereby increasing the amount of information contained in the multiple first keywords. Next, the second core keywords of each environmental information are obtained from the environmental description component. Based on the second core keywords, the main content in the environmental information can be determined. For each environmental information, the second distance between each first keyword and the second core keyword of the environmental information in the knowledge graph is obtained to represent the semantic association degree between the first keyword and the second core keyword. Based on the second distance corresponding to each first keyword, the second association degree between the task action and the environmental information can be determined. The setting of multiple first keywords effectively increases the amount of information, thereby improving the accuracy of the second distance, improving the accuracy of the second association degree, improving the accuracy of the filtered environmental information, and improving the efficiency and accuracy of the large language model task processing method.

[0266] In step 1050, the second correlation degree between task actions and environmental information is determined based on the second distance corresponding to each first keyword.

[0267] In one embodiment, step 1050 includes:

[0268] Step 1110: Input multiple primary keywords into the primary attention model to obtain the primary attention score for each primary keyword;

[0269] Step 1120: Based on the second distance and first attention score corresponding to each first keyword, determine the second correlation between task action and environmental information.

[0270] Steps 1110-1120 are described in detail below.

[0271] In step 1110, multiple first keywords are input into the first attention model to obtain the first attention score for each first keyword.

[0272] The first attention score corresponds to the first keyword. It's a crucial metric in the first attention model for measuring the relevance between different first keywords, quantifying their importance. A higher first attention score indicates greater importance of the first keyword, leading to increased attention paid to it during subsequent UI task processing.

[0273] The first attention model is a model that determines the first attention score for each first keyword. The first attention model can be a transformer model, a pre-trained language model (Bidirectional Encoder Representations from Transformers, BERT), etc.

[0274] Suppose that the primary keywords corresponding to the task action "search for book X" are "bookstore", "search", and "book X", inputting the primary keywords into the primary attention model can yield primary attention scores corresponding to "bookstore", "search", and "book X" respectively.

[0275] In step 1120, the second correlation between task actions and environmental information is determined based on the second distance and first attention score corresponding to each first keyword.

[0276] The second correlation between task actions and environmental information is determined based on the second distance corresponding to each first keyword and the first attention score.

[0277] There are various ways to determine the second degree of relevance, and it is inversely proportional to the second distance and directly proportional to the first attention score. The second degree of relevance can be determined using methods such as table lookup and function mapping. When determining the second degree of relevance using table lookup, the range of the second distance and the range of the first attention score can be divided into multiple intervals. The second distance, the first attention score, and the second degree of relevance correspond to each interval. Therefore, based on the interval containing the first attention score and the interval containing the second distance, the second degree of relevance between the task action and the environmental information can be directly determined.

[0278] In the embodiments of steps 1110-1120 above, the first attention score is taken into account in the process of determining the second correlation degree based on the second distance corresponding to each first keyword. The first attention score is the processing result of the attention model on multiple first keywords. The first attention score can measure the importance of each first keyword. Therefore, the second correlation degree between the task action and the environmental information determined based on the second distance corresponding to the first keyword and the first attention score is more accurate, thereby improving the accuracy of the large language model task processing.

[0279] In step 1120, the second correlation between task actions and environmental information is determined based on the second distance and first attention score corresponding to each first keyword.

[0280] In one embodiment, step 1120 includes:

[0281] Step 1210: Based on the second distance corresponding to the first keyword, determine the second sub-association degree between the first keyword and the environmental information. The second sub-association degree is a decreasing function of the second distance.

[0282] Step 1220: Using the first attention score corresponding to each first keyword, perform a weighted sum operation on the second sub-association degree corresponding to each first keyword to obtain the second association degree between task action and environmental information.

[0283] Steps 1210-1220 are described in detail below.

[0284] In step 1210, based on the second distance corresponding to the first keyword, the second sub-association degree between the first keyword and the environmental information is determined. The second sub-association degree is a decreasing function of the second distance.

[0285] The second sub-association degree is the degree of association between the first keyword and the environmental information, and it is determined based on the second distance. Furthermore, since the degree of association between the first keyword and the environmental information is inversely proportional to the second distance, the second sub-association degree is a decreasing function of the second distance.

[0286] The function between the second sub-association degree and the second distance can be set arbitrarily, as long as the second sub-association degree is a decreasing function of the second distance. This function can be y = 1 - x, y = 1 / x, etc., where y is the second sub-association degree and x is the second distance.

[0287] In step 1220, the second sub-association degree corresponding to each first keyword is weighted and summed based on the first attention score corresponding to each first keyword to obtain the second association degree between task action and environmental information.

[0288] The first attention score represents the importance of the first associated word among multiple first keywords. Therefore, the first attention score can be used as the weight of the first keyword to perform a weighted sum operation on the second sub-associations corresponding to each first keyword, thereby determining the second association between task actions and environmental information.

[0289] It should be noted that the sum of the first attention scores corresponding to each first keyword is 1. If the sum of the first attention scores of each first keyword output by the attention model is not 1, the first attention scores can be transformed to meet the requirement that the sum is 1.

[0290] Suppose that the primary keywords corresponding to the task action "search for book X" are "bookstore", "search", and "book X", where the primary attention score of the primary keyword "bookstore" is 0.3 and the secondary sub-association score is 0.8, the primary attention score of the primary keyword "search" is 0.3 and the secondary sub-association score is 0.5, and the primary attention score of the primary keyword "book X" is 0.4 and the secondary sub-association score is 0.9, then the secondary association score between the task action "search for book X" and the environmental information is 0.75.

[0291] The embodiments of steps 1210-1220 above determine the second sub-association degree between the first keyword and the environmental information based on the first distance corresponding to the first keyword. Then, the second sub-association degree corresponding to each first keyword is weighted and summed using the first attention score corresponding to each first keyword to obtain the second association degree between the task action and the environmental information. Since the first attention score can reflect the importance of the first keyword among multiple first keywords, using the first attention score as the weight in the weighted summation process can improve the accuracy of the result corresponding to the weighted summation, i.e., the second association degree, thereby improving the accuracy of the filtered tool component information and improving the accuracy of the large language model task processing.

[0292] In one embodiment, step 540 includes:

[0293] Step 1310: Obtain the task action type;

[0294] Step 1320: Obtain the task plan from which the task action originates. The task plan is generated based on the task instructions and includes multiple task actions to be executed sequentially.

[0295] Step 1330: Input the task planning into the second attention model to obtain the second attention score of the task action;

[0296] Step 1340: Determine the first threshold based on the task action type and the second attention score;

[0297] Step 1350: Based on the comparison between the second correlation degree and the first threshold, the environmental information is filtered to obtain the filtered environmental information.

[0298] Steps 1310-1350 are described in detail below.

[0299] In step 1310, the task action type of the task action is obtained.

[0300] The task action type is the type of action of the task action. If the task action is "search for book X", then the action type is search. If the task action is "click on product 2", then the action type is product click or product view.

[0301] In step 1320, the task plan from which the task action is derived is obtained. The task plan is generated based on the task instructions and includes multiple task actions to be executed sequentially.

[0302] Task instructions are generated based on the text input by the user. For example, if a user inputs "Calculate the profit of goods this week" into the large language model, the task instruction would be to calculate the profit of goods from [Month X, 2025] to [Month X+7, 2025]. Similarly, if a user inputs "Open book X" into the large language model, the task instruction could be "Open book X".

[0303] Task planning is the planning of multiple tasks generated based on task instructions. Based on task planning, multiple tasks to be executed and the execution order of these tasks can be determined. Assuming the task instruction is "Open book X", the task planning includes the sequentially executed task actions "Search book X" and "Open book X".

[0304] In step 1330, the task planning is input into the second attention model to obtain the second attention score of the task action.

[0305] The second attention score corresponds to the task action and is an important indicator used by the second attention model to measure the correlation between various task actions, quantifying the importance of each action. A higher second attention score indicates greater importance of the task action, leading to increased attention paid to that action during subsequent interface task processing.

[0306] The second attention model is a model that determines the second attention score of task actions. The second attention model can be a transformer model, a pre-trained language model (Bidirectional Encoder Representations from Transformers, BERT), etc.

[0307] Assuming the task planning includes the sequentially executed task actions "search for book X" and "open book X", the task planning is input into the second attention model to obtain the second attention score for the task action "search for book X" and the second attention score for the task action "open book X".

[0308] In step 1340, a first threshold is determined based on the task action type and the second attention score.

[0309] The first threshold is a standard for filtering tool component information, and the first threshold is determined based on the task action type and the second attention score. The first threshold corresponds one-to-one with the task action type.

[0310] In determining the first threshold, the task action type first needs to be converted into a type score. Specifically, the type score corresponding to the task action type can be determined by looking up a table. Then, based on the function corresponding to the type score and the second attention score, the first threshold is determined, and the higher the type score and the second attention score, the higher the first threshold.

[0311] Assume the task planning includes the sequentially executed tasks "Search for book X" and "Open book X". The task action type for "Search for book X" is a search action, and the task action type for "Open book" is a book click action. First, determine the type score for the search action type as 0.6 and the type score for the book click action type as 0.7. Simultaneously, input the task planning into the second attention model, obtaining a second attention score of 0.6 for "Open book X" and 0.4 for "Search for book X". If the first threshold is determined based on the average of the type score and the second attention score, then the first threshold for the task action "Search for book X" is 0.5, and the first threshold for the task action "Open book X" is 0.65.

[0312] It should be noted that the mapping function between the first threshold and the task action type and the second attention score can be set as needed. This mapping function can be the average or product of the type score corresponding to the task action type and the second attention score.

[0313] In step 1350, the environmental information is filtered based on the comparison between the second correlation degree and the first threshold to obtain filtered environmental information.

[0314] After determining the first threshold, for each piece of environmental information in the environmental description component, the second correlation degree corresponding to each piece of environmental information is compared with the first threshold to filter the environmental information, and the environmental information with the second correlation degree greater than the first threshold is taken as the filtered environmental information.

[0315] The embodiments of steps 1310-1350 above filter the environmental information based on the comparison of the first threshold and the second correlation degree of each environmental information to obtain filtered environmental information. The first threshold is determined based on the task action type corresponding to the task action and the second attention score. The second attention score is the processing result of the second attention model on the task planning from which the task action comes. Therefore, the second attention score can determine the importance of the task action in the task planning, and the importance of the action type of the task action can be determined based on the task action type. Thus, the first threshold is more matched with the task action, thereby improving the accuracy of the filtered environmental information and improving the accuracy of the large language model task processing.

[0316] Detailed description of step 430

[0317] In step 430, the tool component invocation instruction is executed to perform the task action.

[0318] In one embodiment, step 430 includes:

[0319] Step 1410: Obtain the sandbox used to execute tool component invocation instructions;

[0320] Step 1420: In the sandbox, execute the tool component invocation command to perform the task action.

[0321] Steps 1410-1420 are described in detail below.

[0322] In step 1410, obtain the sandbox for executing tool component invocation instructions.

[0323] A sandbox is a mechanism for executing tool component invocation instructions. It is a secure isolation mechanism used to execute tool invocation instructions in a controlled environment, thereby ensuring the security of large language models and other program applications. Furthermore, since tool component invocation instructions are executed in the sandbox, access to the global scope, browser interfaces, etc. by the related code of the tool component invocation instructions can be restricted, reducing code conflicts and improving the controllability of tool component invocation instructions.

[0324] In step 1420, within the sandbox, execute the tool component invocation instruction to perform the task action.

[0325] After generating tool component invocation instructions through the large language model, the tool component invocation instructions can be executed in the sandbox to invoke the tool components corresponding to the task actions and execute the task actions.

[0326] The embodiments of steps 1410-1420 above execute tool component invocation instructions in a sandbox to perform task actions. The sandbox setting improves the security, isolation, stability and controllability of the tool component invocation instruction execution process.

[0327] Steps 410-430 have been explained in detail above. The following sections will elaborate on some specific points and related topics. These topics include component package settings, storage hierarchy generation, task planning, and rendering processing.

[0328] Component package settings

[0329] In one embodiment, step 420 includes:

[0330] Step 1510: Obtain the general tool component package information. The general tool component package information is the information of a component package formed by multiple tool components. The component package is shared by multiple task actions.

[0331] Step 1520: Through the knowledge injection component, inject the environment description component, tool component information and general tool component package information corresponding to the target interface into the large language model, so that the large language model can generate tool component call instructions and general tool component package call instructions corresponding to the task action based on the environment information, tool component information and general tool component package information in the environment description component.

[0332] Correspondingly, step 430 includes:

[0333] Step 1530: Execute the tool component invocation instruction and the general tool component package invocation instruction to perform the task action.

[0334] Steps 1510-1530 are described in detail below.

[0335] In step 1510, the general tool component package information is obtained. The general tool component package information is the information of a component package formed by multiple tool components. The component package is shared by multiple task actions.

[0336] A component package contains multiple tool components, and these tool components are related to each other. Furthermore, typically a single tool component corresponds to a single task action, so the component package is shared by multiple task actions.

[0337] The general tool component package information corresponds to the component package and includes detailed information about the component package. It should be noted that the environment description component also includes the environment information corresponding to the component package.

[0338] Figure 7A The component package shown includes a product search component and a product list component. Therefore, the general utility component package information includes a product search component and a product list component. The product search component includes `queryOrderList` (an order query interface that accepts `startTime` and `endTime` parameters), and the product list component can query the product's cost price and selling price using `goodInfo` (a product details interface that accepts the `goodId` parameter). Additionally, the environment description component includes the component package's environment information: "Query details of products sold between the time range `startTime` (start time) and `endTime` (end time)". Furthermore, the component package containing the product search component and the product list component can be shared by tasks such as "Search for product sales on [Date]" and "Query product list".

[0339] In step 1520, the knowledge injection component injects the environment description component, tool component information, and general tool component package information corresponding to the target interface into the large language model, so that the large language model generates tool component call instructions and general tool component package call instructions corresponding to the task actions based on the environment information, tool component information, and general tool component package information in the environment description component.

[0340] After determining the environment description components, tool component information, and general tool component package information corresponding to the target interface, the knowledge injection component injects these components into the large language model. The large language model analyzes and identifies the injected knowledge to determine the tool components and component packages corresponding to the task, and generates tool component invocation instructions for the tool components and general tool component package invocation instructions for the component packages.

[0341] Reference Figure 7A For the task action "calculate this week's profit", the large language model generates general tool component package call instructions for component packages containing product search components and product list components, and tool component call instructions for profit calculation components, based on environment description components, tool component information and general tool component package information.

[0342] It should be noted that if the component package contains multiple tool components corresponding to the task action, the large language model will only generate general tool component package invocation instructions. For example... Figure 7B The component package shown includes a product search component, a product list component, and a profit calculation component. For the task "Calculate this week's profit," the large language model only generates general tool component package invocation instructions to call the product search component, product list component, and profit calculation component. (See reference...) Figure 7C The task action "Query this week's orders" shown in the example only generates a general tool component package invocation instruction to invoke the product search component because the component package includes a product search component.

[0343] It should be noted that in the storage hierarchy, besides the environment description component and tool component information corresponding to the target interface, the information of the general tool component package can be stored separately. Alternatively, after storing the information of the general tool component package, the tool component information of the tool component with the highest degree of association with the tool package can be deleted, for example, referring to... Figure 5B and Figure 7B The information for the general tool component package is stored in the storage hierarchy, replacing the information for the profit calculation component. Alternatively, after storing the general tool component package information, the tool component information for multiple corresponding tool components can be deleted, for example, referring to... Figure 5B and Figure 7AThe general tool component package information is stored in the storage hierarchy instead of the product search component and the product list component.

[0344] In step 1530, tool component invocation instructions and general tool component package invocation instructions are executed to perform task actions.

[0345] After determining the tool component invocation instructions and general tool component package invocation instructions corresponding to the task action, the tool component invocation instructions and general tool component package invocation instructions are executed to realize the execution of the task action.

[0346] Reference Figure 7A After generating general tool component package invocation instructions for the component package containing the product search component and the product list component, as well as tool component invocation instructions for the profit calculation component, using the large language model, the tool component invocation instructions and the general tool component package invocation instructions are executed to perform the task action "Calculate this week's profit".

[0347] Assuming the task is to calculate this week's profit, if Figure 5A The target interface shown is stored in the storage hierarchy as follows: Figure 5B As shown, the large language model needs to sequentially query and determine the product search component, the product list component, and the profit calculation component. However, if Figure 5A The target interface shown is stored in the storage hierarchy as follows: Figure 7A As shown, the large language model only needs to determine the component package and the profit calculation component.

[0348] The embodiments of steps 1510-1530 above include a toolkit containing multiple tool components. The component package is shared by multiple task actions. The storage hierarchy stores the general tool component package information corresponding to the component package. The general tool component package information indicates the association information between multiple tool components. When executing a task action, only the general tool component package call instruction corresponding to the component package needs to be executed. Moreover, the large language model does not need to repeatedly query multiple tool components, which improves the processing efficiency of interface tasks.

[0349] In one embodiment, the generic tool component package is generated in the following manner:

[0350] Step 1610: Assign a component package identifier to the general tool component package;

[0351] Step 1620: Based on the environment information of each of the multiple tool components, generate environment data for the general tool component package;

[0352] Step 1630: Obtain the action type executed by the general tool component package;

[0353] Step 1640: Generate a general tool component package based on the component package identifier, environment data, multiple tool components, and action type.

[0354] Steps 1610-1640 are described in detail below.

[0355] In step 1610, a component package identifier is assigned to the general tool component package.

[0356] The component package identifier is the identifier of a general tool component package. Based on the component package identifier, a unique general tool component package information can be identified in the storage hierarchy, and the corresponding general tool component package can also be invoked.

[0357] It should be noted that general tool component packages are assigned corresponding component package identifiers, and tool components are also assigned corresponding tool component identifiers.

[0358] In step 1620, environmental data for a general tool component package is generated based on the environmental information of each of the multiple tool components.

[0359] A general-purpose tool component package contains multiple tool components; therefore, the environment data for the general-purpose tool component package is generated based on the environment information of each of these tool components. The environment data of the general-purpose tool component package can also be understood as the environment information of the general-purpose tool components. (See reference...) Figure 5B and Figure 7A The product search component's environment information is to query orders located between the time range startTime and endTime, and integrate the query results into an order list. The product list component's environment information is to determine the cost price and selling price of each product based on the order list, and generate a product details list. The general tool component package's environment data is to query the details of products sold between the time range startTime and endTime.

[0360] In step 1630, the action type executed by the general tool component package is obtained.

[0361] Action type refers to the type of action performed by the general tool component package; it can also be considered the task action type of the corresponding task action. Since the general tool component package is shared by multiple task actions, the action type performed by the general tool component package is determined based on the task action types corresponding to those multiple task actions. For example, if the general tool component package contains a product search component and a product list component, and the task action type of the product search component is product search, and the task action type of the product list component is product list query, then the action types performed by the general tool component package are product search and product list query.

[0362] In step 1640, a general tool component package is generated based on the component package identifier, environment data, multiple tool components, and action type.

[0363] The generic tool component package is determined based on the component package identifier, environment data, multiple tool components, and action type. It should be noted that the environment data for the generic tool component package can also be stored within the environment description information.

[0364] The protocol for a generic tool component package can be represented as:

[0365] message MNCSchema{

[0366] string component_id = 1;

[0367] ContextData context = 2;

[0368] Repeated Tool tools = 3;

[0369] ActionType action = 4;

[0370] }

[0371] Among them, stringcomponent_id is the component package identifier, ContextDatacontext is the environment data, repeatedTooltools is a list of multiple tool components in the general tool component package, i.e., the list of available tools, and ActionTypeaction is the action type executed by the general tool component package.

[0372] The embodiments of steps 1610-1640 above generate a general tool component package based on component package identifier, environment data, multiple tool components, and action type. The setting of component package identifier, environment data, multiple tool components, and action type facilitates the large language model in determining the general tool component package call instruction corresponding to the task action, thereby improving the accuracy of the general tool component package call instruction generation. In addition, the environment data, action type, etc., are determined based on the data of multiple tool components in the general tool component package, which further improves the accuracy of each piece of information in the general tool component package, improves the accuracy of the general tool component package call instruction generation, and thus improves the accuracy of the large language model task processing.

[0373] Generation of storage hierarchy

[0374] In one embodiment, prior to step 410, the large language model task processing method further includes:

[0375] Step 1710: Obtain the interface sequence. The interface sequence includes multiple consecutive interfaces obtained by triggering a series of controls starting from the initial interface, and the multiple consecutive interfaces include the target interface.

[0376] Step 1720: For each interface in the interface sequence, obtain the tool component information of the interface and the environment information when calling each tool component in the interface;

[0377] Step 1730: Generate an environment description component based on the environment information when calling each tool component in the interface;

[0378] Step 1740: Store the environment description component and tool component information of the interface according to the position of the interface in the interface training to generate a storage hierarchy structure.

[0379] Steps 1710-1740 are described in detail below.

[0380] In step 1710, an interface sequence is obtained, which includes multiple consecutive interfaces obtained by triggering a series of controls starting from the initial interface, and the multiple consecutive interfaces include the target interface.

[0381] A sequence of interfaces contains multiple interfaces, and the sequence must contain at least the target interface. Other interfaces in the sequence are obtained directly or indirectly based on controls triggered by the starting interface, and the sequence consists of multiple consecutive interfaces.

[0382] Assuming the storage hierarchy corresponds to the XX shopping platform, the starting interface is the main interface of the XX shopping platform, and the interface sequence includes, for example: Figure 2A The image shows the search interface for the mobile phone and the details interface for product 1.

[0383] In step 1720, for each interface in the interface sequence, the tool component information of the interface and the environment information when calling each tool component in the interface are obtained.

[0384] For each interface in the interface sequence, obtain the tool component information of that interface, and the environment information when calling each tool component within that interface. (Refer to...) Figure 5A and Figure 5B Assuming Figure 5A The interface shown is one of the interfaces in the interface sequence. It obtains the tool component information of the interface, namely the tool component information of the product search component, product order component, profit calculation component, etc., and obtains the new environment information when calling each tool component. For example, the environment information of the product search component is to query orders located between the time range startTime and endTime, and integrate the query results into an order list.

[0385] It should be noted that the tool component information of the interface includes not only the tool component information of the multiple tool components that the user can first see in the interface, but also the tool component information related to the domain knowledge corresponding to the interface. For example, refer to... Figure 5A and Figure 5B The profit calculation component is a tool component determined based on the domain knowledge corresponding to the shopping platform interface.

[0386] In addition, the environmental information also includes a description of the entire interface, such as Figure 5B It also includes environment information: "The current interface is the order management interface of the xx shopping platform, which includes a product search component, a product list component, and a profit calculation component."

[0387] In step 1730, an environment description component is generated based on the environment information when calling each tool component in the interface.

[0388] The environment description component is generated based on the environment information when calling various tool components in the interface.

[0389] In step 1740, the environment description component and tool component information of the interface are stored according to the position of the interface in the interface training to generate a storage hierarchy structure.

[0390] Interface training refers to the process of generating a storage hierarchy. After determining the environment description components and tool component information for each interface in the interface sequence, the environment description components and tool component information are stored according to the position of the interface in the interface training, resulting in the storage hierarchy.

[0391] It should be noted that, in order to facilitate the quick determination of the environment information and tool component information corresponding to the tool component, the environment information and tool component information of the tool component can be stored accordingly.

[0392] In the embodiments described in steps 1710-1740 above, during interface training, a series of consecutive interfaces obtained through a series of control triggers, starting from the initial interface, are used as an interface sequence. For each interface in the sequence, the tool component information of the interface and the environment information when calling each tool component in the interface are obtained. Based on the environment information when calling each tool component in the interface, an environment description component can be generated. Then, based on the environment description components and tool component information of each interface, they are stored according to the position of the interface in the interface training, generating a storage hierarchy structure. The setting of the storage hierarchy structure facilitates the rapid determination of the environment description components and tool component information corresponding to the target interface, improving the processing efficiency of large language model tasks.

[0393] Task planning and rendering

[0394] In one embodiment, prior to step 410, the large language model task processing method includes:

[0395] Step 1810: Receive task instructions;

[0396] Step 1820: Based on the task instructions, generate a task plan, the task plan including a plurality of task actions to be executed sequentially;

[0397] Correspondingly, after step 430, the large language model task processing methods include:

[0398] Step 1830: Render the execution result of the task action after it is executed, and in response to rendering the execution result of the last task action after it is executed, obtain the execution result of the task instruction.

[0399] Steps 1810-1830 are described in detail below.

[0400] In step 1810, a task instruction is received.

[0401] Task instructions are generated based on text input by the user. (See reference...) Figure 8A Suppose a user inputs "Help me check the sales profit of goods this week" into the large language model, then the task instruction is to calculate the profit of goods from X month X, 2025 to X month X+7, 2025.

[0402] In step 1820, a task plan is generated based on the task instructions, the task plan including a plurality of task actions to be executed sequentially.

[0403] Task planning is the planning of multiple tasks generated based on task instructions. Based on task planning, multiple tasks to be executed and their execution order can be determined. Assuming the task instruction is to calculate the profit of goods from [Month X, 2025] to [Month X+7, 2025], then the task planning would be as follows: Figure 10A Tasks 1, 2, 3 and 4 are shown.

[0404] In step 1830, the execution result of the task action is rendered, and in response to rendering the execution result of the last task action, the execution result of the task instruction is obtained.

[0405] The action execution result is the outcome of the task actions, and also the result of the tool component's invocation instructions. The action execution results for each task are rendered to display to the user the execution progress, intermediate results, etc., of each task action corresponding to the task instruction. Based on the rendered action execution results, the user can determine the accuracy of the final execution result.

[0406] In addition, after the rendering of the action execution result after the last task action is executed, the execution and rendering of multiple task actions corresponding to the task execution are completed. The action execution result after each task action is executed can be used as the execution result of the task instruction.

[0407] Reference Figure 8B In the dialog box with the smart assistant, the execution result of the task action is displayed, which allows the user to determine the execution result of the task instruction.

[0408] It should be noted that the embodiments of this application may be based on Figure 9 The illustrated process involves task planning and executing multiple actions corresponding to the planned tasks. First, tool retrieval identifies multiple tool components corresponding to the target interface, as well as environment description components and tool component information. Memory control is implemented during task planning to reduce issues such as memory overflow and context loss caused by tool component calls. Next, task planning is performed based on task instructions, resulting in multiple task actions. The large language model generates tool component call instructions corresponding to the task actions based on the injected environment description components and tool component information. These tool component call instructions are then executed to perform the task actions and obtain the action execution results. If the task action is not the last result of task planning, the process jumps to the next task planning step to execute the next task action. Furthermore, after each task action execution, the execution result can be rendered. In response to rendering the execution result after the last executed task action, the execution result of the task instruction is obtained.

[0409] The embodiments described in steps 1810-1830 above generate a task plan for task instructions. The task plan includes multiple task actions to be executed sequentially. Then, the environment description component and tool component information of the target interface can be injected into the large language model, so that the large language model can sequentially determine the tool component invocation instructions for each task action based on the environment description component and tool component information. The execution results of the task actions need to be rendered. After the rendering of the execution result of the last task action is completed, the execution result of the task instruction is obtained. The task planning improves the accuracy of task processing by the large language model, while the rendering of the action execution results visualizes the execution of the task instructions.

[0410] In one embodiment, after step 1810, the large language model task processing method includes:

[0411] Step 1910: Display the task planning panel, which shows the progress of multiple command actions.

[0412] Following step 430, the large language model task processing method includes:

[0413] Step 1920: Display the tool component call status panel. The tool component call status panel displays the status of calling the tool component.

[0414] Correspondingly, step 1830 includes:

[0415] Step 1930: Display the execution results panel and render the action execution results on the execution results panel.

[0416] Steps 1910-1930 are described in detail below.

[0417] In step 1910, the task planning panel is displayed, showing the progress of multiple command actions.

[0418] The task planning panel is the panel that plans the tasks corresponding to the task instructions, and it displays the progress of multiple instruction actions. For the task instruction "Calculate the profit of goods from [Month] [Day], 2025 to [Month] [Day] + 7, 2025", the displayed task planning panel would look like this: Figure 10A As shown, the task planning panel displays four task actions: Task 1, Task 2, Task 3, and Task 4. The task planning panel can highlight the currently executing task action to display the progress of multiple instructions.

[0419] In step 1920, the tool component call status panel is displayed, which shows the status of the called tool component.

[0420] The tool component call status panel is a panel that displays the status of tool components being called during the execution of task instructions. Figure 10B This is a diagram illustrating how a tool component invokes a status panel. The status of the invoked tool component can be determined based on this status panel. (See reference...) Figure 10B It can be confirmed that the product search component has been successfully invoked and the corresponding task has been completed. The product list component is currently being invoked, while the profit calculation component is waiting to be invoked.

[0421] It should be noted that, Figure 10B The tool component call status panel also displays the call code for each tool component. Assuming the product search tool component is called, the code displayed in the tool component call status panel would be as follows:

[0422]

[0423]

[0424] Here, `constresources = awaitclient.queryOrderList()` represents the product search component, `constresult = awaitclient.callTool` represents the call to the product search component, and `startTime` and `endTime` are two parameters that the search component needs to receive.

[0425] In step 1930, the execution results panel is displayed, and the action execution results are rendered on the execution results panel.

[0426] The Execution Results panel displays the results of the action execution. Therefore, after obtaining the execution results of the task actions, the results can be rendered on the Execution Results panel.

[0427] It should be noted that the large language model task processing method can also display a data analysis panel. For the task instruction "Calculate the commodity profit from [Month] [Day], 2025 to [Month] [Day]+7, 2025", the following can be displayed: Figure 10C The data analysis panel shown above allows you to determine the sales volume, profit, percentage of total profit, and profit margin of each product.

[0428] Reference Figure 8A User A can open the smart window mode and enter the text "Help me check this week's product sales profit" in the smart assistant's dialog box. Afterwards, the target interface is dynamically compressed to approximately 2 / 3 of its size, and simultaneously, according to the task command's execution progress, the task planning panel, tool component call status panel, and data analysis panel are displayed sequentially. This interface is as follows: Figure 10D As shown.

[0429] The embodiments of steps 1910-1930 above include a task planning panel, a tool component call status panel, and an execution result panel. The task planning panel displays the progress of multiple instruction actions, the tool component call status panel displays the status of calling tool components, and the execution result panel displays the action execution results of each task action. The task planning panel, tool component call status panel, and execution result panel realize the visualization of the task instruction execution process.

[0430] In one embodiment, the large language model task processing method of this disclosure is implemented through... Figure 11The illustrated edge-side intelligent enhancement system implementation, starting from the bottom layer, consists of an infrastructure layer, a storage hierarchy, a protocol adaptation layer, a knowledge injection component, and an intelligent application layer. The infrastructure layer primarily includes a sandbox, used for executing tool component invocation commands and rendering. The storage hierarchy contains environment description components and tool component information, assisting the large language model in understanding task actions and performing high-precision operations. Furthermore, all tool components conform to the model context protocol and can be injected into the large language model through the knowledge injection system. The protocol adaptation layer connects the context content injected by the knowledge injection layer, i.e., the environment description component and tool component information, and converts it into a fixed protocol for transmission to the large language model. It also performs data reclamation and processing by invoking tool components. The intelligent application layer corresponds to... Figure 8A Smart assistants in China, and Figure 10D The smart window in the game.

[0431] In one embodiment, reference is made to Figure 12 , Figures 10A-10D and Figure 8A After enabling smart window mode, Figure 8A The right-hand page dynamically compresses by 2 / 3, then displays a task planning panel showing multiple task actions corresponding to the task instructions, i.e., Tasks 1 to 4. Each task action is then executed. For Task 1, "Get the list of products from the past week," the product search component `queryOrderList` is called, passing in the start time and end time parameters. The query results are recorded as an order list `orderList` and stored in memory. For Task 2, "Get the cost price, selling price, and commission of products," based on the order list `orderList`, the product list component `goodInfo` is called in batches, passing in the product identifier `goodId` parameter, generating a product details table `goodsDetailList`. For Task 3, "Calculate the profit formula," the product details table `goodsDetailList` is traversed, retrieving the cost price `costPrice` and selling price `sellingPrice` fields, passing them to the profit calculation component `profitCalculation`, and accumulating the execution results to obtain the final profit. The profit data is recorded as `proFitData`. During data analysis, a query is performed to see if there is an icon component matching the profit data. If so, the icon component is called and rendered; otherwise, an HTML icon is generated and rendered. Figure 12 The execution of each task action and the invocation of tool components are all in a directed acyclic graph pattern, that is, the execution of task actions and the invocation of tool components can be carried out in any order, as long as the tool component can receive the corresponding parameters.

[0432] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0433] It should be noted that in various specific embodiments of this application, when processing data related to object characteristics, such as object attribute information or sets of attribute information, is required, the object's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining object attribute information, separate permission or consent from the object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the object's separate permission or consent will the necessary object-related data for the proper functioning of these embodiments be acquired.

[0434] Description of apparatus and devices according to embodiments of this disclosure

[0435] Figure 13 A schematic diagram of the structure of a large language model task processing device 2000 provided in an embodiment of this disclosure. The large language model task processing device 2000 includes:

[0436] The first acquisition unit 2010 is used to acquire the environment description component corresponding to the target interface and the tool component information of the target interface from the storage hierarchy structure in response to the task action on the target interface. The storage hierarchy structure stores the environment description component and tool component information according to the interface. The environment description component contains the environment information when calling each tool component in the interface.

[0437] The injection unit 2020 is used to inject the environment description component and tool component information corresponding to the target interface into the large language model through the knowledge injection component, so that the large language model can generate tool component call instructions corresponding to the task action based on the environment information in the environment description component and the tool component information.

[0438] The execution unit 2030 is used to execute tool component invocation instructions to perform task actions.

[0439] Optionally, the large language model task processing device 2000 also includes:

[0440] The second acquisition unit is used to acquire the interface sequence, which includes multiple consecutive interfaces obtained from the starting interface through a series of control triggers, and the multiple consecutive interfaces include the target interface.

[0441] The third acquisition unit is used to acquire tool component information of each interface in the interface sequence, and environment information when calling each tool component in the interface;

[0442] The first generation unit is used to generate an environment description component based on the environment information when calling various tool components in the interface;

[0443] The storage unit is used to store the environment description component and tool component information of the interface according to the position of the interface in the interface training, so as to generate a storage hierarchy structure.

[0444] Optionally, the large language model task processing device 2000 also includes:

[0445] The receiving unit is used to receive task instructions;

[0446] The second generation unit is used to generate a task plan based on the task instructions. The task plan includes multiple task actions to be executed sequentially.

[0447] The rendering unit is used to render the execution result of the task action after it is executed, and in response to rendering the execution result of the last task action, it obtains the execution result of the task instruction.

[0448] Optionally, the large language model task processing device 2000 also includes:

[0449] The first display unit is used to display the task planning panel, which shows the progress of the execution of multiple command actions;

[0450] The second display unit is used to display the tool component call status panel, which displays the status of calling the tool component.

[0451] The rendering unit is specifically used to: display the execution results panel and render the action execution results on the execution results panel.

[0452] Optionally, the injection unit 2020 is specifically used for:

[0453] Obtain information about the general tool component package. This information is from a component package formed by multiple tool components, and the component package is shared by multiple task actions.

[0454] Through the knowledge injection component, the environment description component, tool component information and general tool component package information corresponding to the target interface are injected into the large language model, so that the large language model can generate tool component call instructions and general tool component package call instructions corresponding to the task actions based on the environment information, tool component information and general tool component package information in the environment description component;

[0455] The execution unit 2030 is specifically used to execute tool component invocation instructions and general tool component package invocation instructions to perform task actions.

[0456] Optionally, the large language model task processing device 2000 also includes:

[0457] The allocation unit is used to assign component package identifiers to general tool component packages;

[0458] The third generation unit is used to generate environment data for a general tool component package based on the environment information of each of the multiple tool components.

[0459] The fourth acquisition unit is used to acquire the action type executed by the general tool component package;

[0460] The fourth generation unit is used to generate a general tool component package based on the component package identifier, environment data, multiple tool components, and action type.

[0461] Optionally, the injection unit 2020 is also specifically used for:

[0462] Based on task actions and tool component information, determine the first degree of association between the task actions and each tool component in the target interface;

[0463] The tool component information is filtered based on the first degree of relevance to obtain the filtered tool component information;

[0464] Based on the task action and environment description components, determine the second degree of correlation between the task action and each piece of environmental information;

[0465] The environmental information is filtered based on the second degree of correlation to obtain the filtered environmental information.

[0466] The knowledge injection component injects filtered tool component information and filtered environment information into the large language model, enabling the large language model to generate tool component invocation instructions corresponding to task actions based on the filtered tool component information and filtered environment information.

[0467] Optionally, the injection unit 2020 is also specifically used for:

[0468] Input the task action into the background augmentation model to obtain the first background description of the task action;

[0469] Extract multiple primary keywords from the primary background description;

[0470] Extract the primary core keyword for each tool component from the tool component information;

[0471] For each tool component, obtain the first distance between each primary keyword and the primary core keyword of the tool component in the knowledge graph;

[0472] Based on the first distance corresponding to each first keyword, the first degree of association between the task action and the tool component is determined.

[0473] Optionally, the injection unit 2020 is also specifically used for:

[0474] Input multiple primary keywords into the primary attention model to obtain the primary attention score for each primary keyword;

[0475] Based on the first distance and first attention score corresponding to each first keyword, the first degree of association between the task action and the tool component is determined.

[0476] Optionally, the injection unit 2020 is also specifically used for:

[0477] Based on the first distance corresponding to the first keyword, the first sub-association degree between the first keyword and the tool component is determined. The first sub-association degree is a decreasing function of the first distance.

[0478] The first attention score corresponding to each first keyword is used to calculate the first sub-association degree corresponding to each first keyword by weighting and summing, so as to obtain the first association degree between the task action and the tool component.

[0479] Optionally, the injection unit 2020 is also specifically used for:

[0480] Get the task action type of the task action;

[0481] Obtain the task plan from which the task actions originate. The task plan is generated based on the task instructions and includes multiple task actions to be executed sequentially.

[0482] Input the task planning into the second attention model to obtain the second attention score of the task action;

[0483] The first threshold is determined based on the task action type and the second attention score;

[0484] Based on the comparison between the first correlation degree and the first threshold, the tool component information is filtered to obtain the filtered tool component information.

[0485] Optionally, the injection unit 2020 is also specifically used for:

[0486] Input the task action into the background augmentation model to obtain the first background description of the task action;

[0487] Extract multiple primary keywords from the primary background description;

[0488] Extract the second core keywords for each environmental information from the environmental description component;

[0489] For each piece of environmental information, obtain the second distance between each primary keyword and the second core keyword of the environmental information in the knowledge graph;

[0490] Based on the second distance corresponding to each first keyword, the second degree of correlation between task actions and environmental information is determined.

[0491] Optionally, the injection unit 2020 is also specifically used for:

[0492] Input multiple primary keywords into the primary attention model to obtain the primary attention score for each primary keyword;

[0493] Based on the second distance and first attention score corresponding to each first keyword, the second correlation between task actions and environmental information is determined.

[0494] Optionally, the injection unit 2020 is also specifically used for:

[0495] Based on the second distance corresponding to the first keyword, the second sub-association degree between the first keyword and the environmental information is determined. The second sub-association degree is a decreasing function of the second distance.

[0496] The second sub-association degree corresponding to each first keyword is weighted and summed based on the first attention score corresponding to each first keyword to obtain the second association degree between task action and environmental information.

[0497] Optionally, the injection unit 2020 is also specifically used for:

[0498] Get the task action type of the task action;

[0499] Obtain the task plan from which the task actions originate. The task plan is generated based on the task instructions and includes multiple task actions to be executed sequentially.

[0500] Input the task planning into the second attention model to obtain the second attention score of the task action;

[0501] The first threshold is determined based on the task action type and the second attention score;

[0502] Based on the comparison between the second correlation degree and the first threshold, the environmental information is filtered to obtain the filtered environmental information.

[0503] Reference Figure 14 , Figure 14 To implement the structural block diagram of the terminal portion of the large language model task processing method according to the embodiments of this disclosure, the terminal includes: a radio frequency (RF) circuit 2110, a memory 2115, an input unit 2130, a display unit 2140, a sensor 2150, an audio circuit 2160, a wireless fidelity (WiFi) module 2170, a processor 2180, and a power supply 2190, etc. Those skilled in the art will understand that... Figure 14 The terminal structure shown does not constitute a limitation on mobile phones or computers and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0504] The RF circuit 2110 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 2180; in addition, it transmits uplink data to the base station.

[0505] The memory 2115 can be used to store software programs and modules, and the processor 2180 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 2115.

[0506] The input unit 2130 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the content terminal. Specifically, the input unit 2130 may include a touch panel 2131 and other input devices 2132.

[0507] Display unit 2140 can be used to display input or provided information, as well as various menus of the content terminal. Display unit 2140 may include display panel 2141.

[0508] Audio circuitry 2160, speaker 2161, and microphone 2162 provide an audio interface.

[0509] In this embodiment, the processor 2180 included in the terminal can execute the large language model task processing method of the previous embodiment.

[0510] The terminals disclosed in this embodiment include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this invention can be applied to various scenarios, including but not limited to cloud application data transmission.

[0511] Figure 15This is a partial structural block diagram of a server for implementing the large language model task processing method of this disclosure. The server can vary significantly due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 2222 (e.g., one or more processors) and memory 2232, and one or more storage media 2230 (e.g., one or more mass storage devices) for storing application programs 2242 or data 2244. The memory 2232 and storage media 2230 can be temporary or persistent storage. The program stored in the storage media 2230 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 2222 may be configured to communicate with the storage media 2230 and execute the series of instruction operations in the storage media 2230 on the server.

[0512] The server may also include one or more power supplies 2226, one or more wired or wireless network interfaces 2250, one or more input / output interfaces 2258, and / or one or more operating systems 2241, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, etc.

[0513] The central processing unit 2222 in the server can be used to execute the large language model task processing method of the present disclosure embodiments.

[0514] This disclosure also provides a computer-readable storage medium for storing program code for executing the large language model task processing methods of the foregoing embodiments.

[0515] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the large language model task processing method described above.

[0516] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar terms and are not necessarily used to describe a particular order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0517] It should be understood that in this disclosure, "at least one item" refers to one or more items, and "more than one item" refers to two or more items. "And / or" is used to describe the relationship between related content, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related content are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0518] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0519] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0520] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0521] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0522] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server 130, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0523] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0524] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A method for processing large language model tasks, characterized in that, include: In response to a task action on a target interface, the environment description component corresponding to the target interface and the tool component information of the target interface are obtained from the storage hierarchy. The storage hierarchy stores the environment description component and the tool component information by interface. The environment description component contains environment information when calling each tool component in the interface. The knowledge injection component injects the environment description component and tool component information corresponding to the target interface into the large language model, so that the large language model generates the tool component invocation instruction corresponding to the task action based on the environment information in the environment description component and the tool component information. Execute the tool component invocation instruction to perform the task action.

2. The method according to claim 1, characterized in that, Before retrieving the environment description component corresponding to the target interface and the tool component information of the target interface from the storage hierarchy in response to a task action on the target interface, the method further includes: Obtain an interface sequence, the interface sequence comprising multiple consecutive interfaces obtained by triggering a series of controls starting from a starting interface, and the multiple consecutive interfaces including the target interface; For each interface in the interface sequence, obtain the tool component information of the interface and the environment information when calling each tool component in the interface; Based on the environment information when calling each tool component in the interface, the environment description component is generated; The environment description component and tool component information of the interface are stored according to the position of the interface in the interface training to generate the storage hierarchy structure.

3. The method according to claim 1, characterized in that, Before retrieving the environment description component corresponding to the target interface and the tool component information of the target interface from the storage hierarchy in response to a task action on the target interface, the method further includes: Receive task instructions; Based on the task instructions, a task plan is generated, which includes multiple task actions to be executed sequentially. After executing the tool component invocation instruction to perform the task action, the method further includes: rendering the action execution result after the task action is executed, and in response to rendering the action execution result after the last task action is executed, obtaining the execution result of the task instruction.

4. The method according to claim 3, characterized in that, After generating a task plan based on the task instructions, the method further includes: displaying a task planning panel, wherein the task planning panel displays the progress of the execution of multiple instruction actions; After executing the tool component invocation instruction to perform the task action, the method further includes: displaying a tool component invocation status panel, the tool component invocation status panel displaying the status of invoking the tool component; The rendering of the action execution result after the task action is executed includes: displaying the execution result panel and rendering the action execution result on the execution result panel.

5. The method according to claim 1, characterized in that, The step of injecting the environment description component and tool component information corresponding to the target interface into the large language model through the knowledge injection component, so that the large language model generates the tool component invocation instruction corresponding to the task action based on the environment information in the environment description component and the tool component information, includes: Obtain general tool component package information, which is information about a component package formed by multiple tool components, and the component package is shared by multiple task actions; The knowledge injection component injects the environment description component, tool component information, and general tool component package information corresponding to the target interface into the large language model, so that the large language model generates the tool component call instruction and the general tool component package call instruction corresponding to the task action based on the environment information, tool component information, and general tool component package information in the environment description component. The step of executing the tool component invocation instruction to perform the task action includes: executing the tool component invocation instruction and the general tool component package invocation instruction to perform the task action.

6. The method according to claim 5, characterized in that, The general tool component package is generated in the following way: Assign a component package identifier to the general tool component package; Based on the environmental information of each of the multiple tool components, the environmental data of the general tool component package is generated; Obtain the action type executed by the general tool component package; The general tool component package is generated based on the component package identifier, the environment data, the multiple tool components, and the action type.

7. The method according to claim 1, characterized in that, The execution of the tool component invocation instruction to perform the task action includes: Obtain a sandbox for executing the tool component invocation instructions; Within the sandbox, the tool component invocation instructions are executed to perform the task actions.

8. The method according to claim 1, characterized in that, The step of injecting the environment description component and tool component information corresponding to the target interface into the large language model through the knowledge injection component, so that the large language model generates the tool component invocation instruction corresponding to the task action based on the environment information in the environment description component and the tool component information, includes: Based on the task action and the tool component information, determine the first degree of association between the task action and each tool component in the target interface; The tool component information is filtered based on the first correlation to obtain filtered tool component information; Based on the task action and the environment description component, a second degree of correlation between the task action and each piece of environmental information is determined; The environmental information is filtered based on the second correlation to obtain filtered environmental information. The knowledge injection component injects the filtered tool component information and the filtered environment information into the large language model, enabling the large language model to generate the tool component invocation instruction corresponding to the task action based on the filtered tool component information and the filtered environment information.

9. The method according to claim 8, characterized in that, The step of determining the first correlation degree between the task action and each tool component in the target interface based on the task action and the tool component information includes: The task action is input into the background augmentation model to obtain the first background description of the task action; Extract multiple first keywords from the first background description; From the tool component information, obtain the first core keyword of each tool component; For each tool component, obtain the first distance between each first keyword and the first core keyword of the tool component in the knowledge graph; Based on the first distance corresponding to each of the first keywords, the first correlation degree between the task action and the tool component is determined.

10. The method according to claim 9, characterized in that, Determining the first correlation between the task action and the tool component based on the first distance corresponding to each of the first keywords includes: Input the multiple first keywords into the first attention model to obtain the first attention score for each first keyword; Based on the first distance and the first attention score corresponding to each of the first keywords, the first correlation between the task action and the tool component is determined.

11. The method according to claim 10, characterized in that, Determining the first correlation between the task action and the tool component based on the first distance and the first attention score corresponding to each of the first keywords includes: Based on the first distance corresponding to the first keyword, a first sub-association degree between the first keyword and the tool component is determined, wherein the first sub-association degree is a decreasing function of the first distance; The first attention score corresponding to each first keyword is used to perform a weighted sum operation on the first sub-association degree corresponding to each first keyword to obtain the first association degree between the task action and the tool component.

12. The method according to claim 8, characterized in that, The filtering of the tool component information based on the first correlation degree to obtain filtered tool component information includes: Obtain the task action type of the task action; Obtain the task plan from which the task action originates, the task plan being generated based on task instructions and including multiple task actions to be executed sequentially; The task plan is input into the second attention model to obtain the second attention score of the task action; Based on the task action type and the second attention score, a first threshold is determined; Based on the comparison between the first correlation degree and the first threshold, the tool component information is filtered to obtain the filtered tool component information.

13. The method according to claim 8, characterized in that, The step of determining the second correlation degree between the task action and various environmental information based on the task action and the environment description component includes: The task action is input into the background augmentation model to obtain the first background description of the task action; Extract multiple first keywords from the first background description; The second core keywords for each environmental information are obtained from the environmental description component. For each piece of environmental information, obtain the second distance between each first keyword and the second core keyword of the environmental information in the knowledge graph; Based on the second distance corresponding to each of the first keywords, the second correlation degree between the task action and the environmental information is determined.

14. The method according to claim 13, characterized in that, The step of determining the second correlation degree between the task action and the environmental information based on the second distance corresponding to each of the first keywords includes: Input the multiple first keywords into the first attention model to obtain the first attention score for each first keyword; Based on the second distance and the first attention score corresponding to each of the first keywords, the second correlation between the task action and the environmental information is determined.

15. The method according to claim 14, characterized in that, The step of determining the second correlation between the task action and the environmental information based on the second distance corresponding to each of the first keywords and the first attention score includes: Based on the second distance corresponding to the first keyword, a second sub-association degree between the first keyword and the environmental information is determined, wherein the second sub-association degree is a decreasing function of the second distance; The second sub-association degree corresponding to each of the first keywords is weighted and summed based on the first attention score corresponding to each of the first keywords to obtain the second association degree between the task action and the environmental information.

16. The method according to claim 8, characterized in that, The step of filtering the environmental information based on the second correlation to obtain filtered environmental information includes: Obtain the task action type of the task action; Obtain the task plan from which the task action originates, the task plan being generated based on task instructions and including multiple task actions to be executed sequentially; The task plan is input into the second attention model to obtain the second attention score of the task action; Based on the task action type and the second attention score, a first threshold is determined; Based on the comparison between the second correlation degree and the first threshold, the environmental information is filtered to obtain the filtered environmental information.

17. A large language model task processing device, characterized in that, include: The first acquisition unit is configured to, in response to a task action on the target interface, acquire from the storage hierarchy the environment description component corresponding to the target interface and the tool component information of the target interface, wherein the storage hierarchy stores the environment description component and the tool component information by interface, and the environment description component contains environment information when calling each tool component in the interface; An injection unit is used to inject the environment description component and the tool component information corresponding to the target interface into the large language model through a knowledge injection component, so that the large language model generates a tool component invocation instruction corresponding to the task action based on the environment information in the environment description component and the tool component information. An execution unit is used to execute the tool component invocation instructions to perform the task action.

18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the large language model task processing method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the large language model task processing method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the large language model task processing method according to any one of claims 1 to 16.

Images