Method and system for guiding large-scale language models for task planning and execution

Abstract

Disclosed is a method for guiding large-scale language models (LLMs) for task planning and execution, the method comprising creating a playbook database (302) containing a set of playbooks (SOPs), obtaining a user query (308) received by a large-scale language model (LLM) (306, 208), and determining whether a given task (GT) matches a corresponding descriptor in one of the playbooks among the SOPs, wherein if it is determined that a given task (GT) matches a given descriptor in a given playbook (GP) (314, 400) among the SOPs, the LLM selects one or more external tools (ETs) (212, 316), a software framework associated with the LLM works with the LLM to use the one or more external tools to track and execute a given set of subtasks in the given playbook, the LLM generates a query response containing the results of executing the given set of subtasks, and sends the query response to the user device.

Description

Technical Field

[0001] The present disclosure relates to a method for guiding large language models (LLMs) for task planning and execution. Furthermore, the present disclosure relates to a system for guiding large language models (LLMs) for task planning and execution. Still further, the present disclosure relates to a computer program product.

Background Art

[0002] In recent years, large language models (LLMs) have emerged as a transformative advance in the field of natural language processing (NLP). LLMs are a type of artificial intelligence model designed to process and generate human language, and LLMs are trained on massive volumes of text data to learn the patterns, structures and nuances of human language. LLMs designed to understand and generate human language demonstrate remarkable capabilities in performing a wide range of computational tasks.

[0003] However, the reasoning capabilities of LLMs have limitations, which hinder the ability of LLMs to perform computational tasks. Numerous solutions have been applied to LLMs to improve their reasoning capabilities. In some solutions, LLMs are equipped with the ability to generate plans for improving responses generated from the LLMs for execution of computational tasks and to use external software tools. However, for highly complex computational tasks that require multi-step analysis, responses generated from LLMs to perform the highly complex computational tasks are still sometimes inconsistent, incomplete and inaccurate. Therefore, in view of the above discussion, there exists a need to overcome the aforementioned drawbacks.

Summary of the Invention

Means for Solving the Problems

[0004] The purpose of this disclosure is to provide methods, systems, and computer program products for obtaining accurate and consistent results in the execution of tasks. This purpose is achieved by methods, systems, and computer program products for guiding large-scale language models (LLMs) for task planning and execution as defined in the attached independent claims; please refer to the attached independent claims. Advantageous features are described in the attached dependent claims.

[0005] Throughout this specification and its claims, “comprise,” “include,” “have,” and “contain,” as well as variations thereof, such as “comprising” and “comprises,” mean “including, but not limited to,” and do not preclude the existence of other components, items, elements, or processes not expressly disclosed. Furthermore, unless the context requires otherwise, the singular form also includes the plural form. In particular, where the indefinite article is used, unless the context requires otherwise, this specification should be understood to assume both singular and plural forms. [Brief explanation of the drawing]

[0006] [Figure 1] Figures 1A and 1B are flowcharts illustrating the steps of a method for inducing large-scale language models (LLMs) for task planning and execution, according to one embodiment of the present disclosure. [Figure 2] A diagram showing a block diagram of a system for guiding large-scale language models (LLMs) for task planning and execution, according to one embodiment of the present disclosure. [Figure 3] A flowchart illustrating an implementation scenario for a system that guides large-scale language models (LLMs) for task planning and execution, according to one embodiment of the present disclosure. [Figure 4] A diagram showing a flowchart illustrating the execution of a set of corresponding subtasks in a playbook in a graphical order, according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0007] The following detailed description illustrates embodiments of the Disclosure and methods for carrying them out. While several forms of carrying out the Disclosure are disclosed, those skilled in the art will recognize that other embodiments for carrying out or practicing the Disclosure are also possible.

[0008] In a first embodiment, the Disclosure provides a method for guiding large-scale language models (LLMs) for task planning and execution, the method being: Create a playbook database containing a set of playbooks, each playbook having a corresponding descriptor indicating the task that guides its execution, and containing a plaintext description corresponding to a set of subtasks to be performed for the execution of that task. The Large-Scale Language Model (LLM) retrieves a user query, which concerns a request to perform a given task. The given task determines whether it matches the corresponding descriptor of any of the playbooks in the set of playbooks. If it is determined that the given task matches a given descriptor of a given playbook in the set of playbooks, The LLM selects one or more external tools, and the one or more external tools read the plaintext descriptions of a given set of subtasks. The software framework associated with the LLM, in cooperation with the LLM, uses one or more external tools to track and execute the given set of subtasks in the given playbook. The LLM includes generating a query response containing the results of executing the given set of subtasks and sending the query response to the user device.

[0009] This method significantly improves the consistency and accuracy of the second response generated as a result of executing a given task. This improvement in consistency and accuracy is achieved because the execution of the given task is guided by a given playbook. This ensures that the execution of a given set of subtasks in a given playbook is performed accurately and completely, thus guaranteeing that the steps for executing the given task are performed correctly and completely.

[0010] In a second aspect, the Disclosure provides a system for guiding large-scale language models (LLMs) for task planning and execution, the system, Data repository and At least one processor communicatively coupled to the data repository, wherein the at least one processor is configured to execute a software framework. The aforementioned software framework is In the aforementioned data repository, a playbook database is created containing a set of playbooks, each playbook having a corresponding descriptor indicating a task that guides its execution, and containing a plaintext description of a corresponding set of subtasks to be executed for the execution of that task. The Large-Scale Language Model (LLM) retrieves user queries received, and these user queries relate to requests to execute a given task. The given task is configured to perform a determination of whether it matches a corresponding descriptor in any of the playbooks in the set of playbooks. If it is determined that the given task matches a given descriptor of a given playbook in the set of playbooks, The LLM selects one or more external tools to perform the given task. The software framework associated with the LLM, in cooperation with the LLM, uses one or more external tools to track and execute a given set of subtasks in the given playbook. The LLM generates a query response containing the results of executing the given set of subtasks and sends the query response to the user device.

[0011] The system used to implement the method described above significantly improves the consistency and accuracy of a second response generated as a result of the execution of a given task. The consistency and accuracy of the second response are improved by the fact that the execution of the given task is guided by a given playbook. This ensures that the exact and complete steps for the execution of the given task are performed by executing a given set of subtasks in a given playbook.

[0012] In a third embodiment, the Disclosure provides a computer program product comprising a non-temporary computer-readable storage medium, wherein computer-readable instructions are stored on the computer-readable storage medium, and the computer-readable instructions are executable by at least one processor configured to perform the method of the first embodiment.

[0013] Throughout this disclosure, the term “Large-Scale Language Model” (LLM) refers to a type of artificial intelligence model designed to process, understand, and generate human language. Known large-scale language models can be used in this invention. In particular, LLMs are capable of learning the patterns, structures, and nuances of human language, thereby enabling them to develop a high level of understanding of human language and generate logical text in the form of human language. Since LLMs rely on the use of deep learning techniques, such as the use of transformers that excel at understanding long-range dependencies in any sequence of data, it will be understood that LLMs are better suited to the advanced processing, understanding, and generation of human language. Therefore, LLMs are suitable for use in task planning and execution related to various technical fields, because the task planning and execution related to these various technical fields require the generation of text in human language, translation, question answering, etc., which can be made possible by the use of LLMs. In this regard, the method is implemented to guide or direct LLMs for task planning and execution through a stepwise process that ensures more accurate and consistent results.

[0014] The method involves creating a playbook database containing a set of playbooks, each playbook having a corresponding descriptor indicating a task that guides its execution, and including a plaintext description of a set of corresponding subtasks to be performed for the execution of that task. Throughout this disclosure, the term “playbook” refers to a data structure (e.g., in the form of a computer-readable file) created to contain information about a plan that describes the necessary steps to be followed and performed for the execution of the task that guides its execution. It will be understood that the information is stored within each playbook in the form of digital text written in a human language. In particular, the information stored within each playbook includes information about the corresponding descriptor indicating a task that guides its execution, and information about the plaintext description of a set of corresponding subtasks to be performed for the execution of that task.

[0015] Throughout this disclosure, the term "corresponding descriptor" refers to an attribute or feature that makes it possible to determine which specific task a specific playbook is guiding the execution of. In particular, the corresponding descriptor plays an important role in distinguishing each playbook from all other playbooks present in the set of playbooks. Optionally, the corresponding descriptor of each playbook is stored in the form of text within each playbook. Optionally, the corresponding descriptor of each playbook may be the corresponding title of that playbook stored in the form of text within each playbook. For example, the text of the corresponding descriptor of an exemplary playbook is "Identify why the website is not responding, not working, or not loading as expected", which is stored within the exemplary playbook.

[0016] Throughout this disclosure, the term "corresponding set of subtasks" refers to one or more intermediate tasks that should be performed in a predetermined manner to enable each playbook to guide the corresponding task. In particular, each subtask in the corresponding set of subtasks is a corresponding computational operation to be performed. Throughout this disclosure, the term "plaintext description" refers to a text description written in human natural language that describes the execution method of each subtask and the external tools required to execute each subtask in the corresponding set of subtasks. In other words, the plaintext description is a text description of the corresponding set of subtasks. Therefore, the corresponding set of subtasks can be easily created. Furthermore, the corresponding set of subtasks is suitable for being read, processed and understood by an LLM.

[0017] For example, the plaintext description of the corresponding set of subtasks is as follows. "1. Verify that DNS is correctly configured for the target domain and returns a valid IP address. 2. Verify that each IP address is open on ports 80 and 443. 3. If both ports are open, use the curl command to verify that the open ports are returning valid HTML pages and that there are no problems with the SSL configuration via HTTPS. 4. If only one port is open, use the curl command to verify that the open port is returning valid HTML. Note that this is not typical. 5. If none of the web ports are open, this is likely the problem. 6. If everything appears normal, as a final check, perform Step 3 from each continental location to verify that there are no routing issues and that all responses are similar.

[0018] Throughout the present disclosure, the term "playbook database" refers to a database created for the purpose of storing a set of playbooks in an organized manner. Optionally, the playbook database may be a folder containing files corresponding to the set of playbooks. In particular, creating the playbook database provides an available library that allows a given playbook in the set of playbooks to be accessed when needed to guide the execution of a given task. Optionally, the playbook database may be created using software capable of generating text files. Optionally, the playbook database may be created within a GitHub® repository. It will be appreciated that there are two ways to add any playbook to the playbook database. The first is a method of adding a text file corresponding to any playbook to the playbook database (e.g., within a GitHub® repository). The second is a method of allowing a user to add any playbook to the playbook database via a chat interface of a software framework on a user device when a suitable playbook is not found in the playbook database. Throughout the present disclosure, the term "set of playbooks" refers to one or more playbooks stored in the playbook database, and each playbook in the set of playbooks is created to guide the execution of a specific task.

[0019] Optionally, the step of creating the playbook database includes for a plurality of playbooks, receiving a plaintext description of the corresponding descriptor and the corresponding set of subtasks; performing at least one validity check on the plurality of playbooks; determining whether at least one of the plurality of playbooks passes the at least one validity check; If it is determined that at least one of the plurality of playbooks passes the at least one validation check, the at least one playbook is added to the playbook database.

[0020] In this context, the term “multiple playbooks” refers to multiple different playbooks that guide the execution of different tasks related to various technical fields, and the correct and appropriate playbooks are selected from among the multiple playbooks and added to the playbook database as part of a set of playbooks. In particular, the plaintext descriptions of corresponding descriptors and corresponding subtask sets for the multiple playbooks are received from the user via a user device or from a database that stores the plaintext descriptions of corresponding descriptors and corresponding subtask sets for the multiple playbooks. It will be understood that receiving the plaintext descriptions of corresponding descriptors and corresponding subtask sets for the multiple playbooks ensures that sufficient information is available to add at least one of the multiple playbooks to the playbook database. Throughout this disclosure, the term “at least one validation check” refers to one or more tests performed on the multiple playbooks to verify the quality and accuracy of the multiple playbooks.

[0021] In this regard, to verify the accuracy of multiple playbooks, at least one validation check is used to verify how identifiable and unambiguous the corresponding descriptors of each playbook among the multiple playbooks are compared to the corresponding descriptors of the set of playbooks in the playbook database. Similarly, to verify the quality of multiple playbooks, at least one validation check is used to verify the quality of the plaintext descriptions of the corresponding set of subtasks of each playbook among the multiple playbooks. In particular, by performing at least one validation check on multiple playbooks, it becomes possible to determine at least one of multiple playbooks that is suitable for addition to the playbook database. Throughout this disclosure, the term “at least one of multiple playbooks” refers to one or more playbooks that are suitable for addition to the playbook database because they have passed at least one validation check. It will be understood that determining at least one of multiple playbooks that pass at least one validation check indicates that at least one playbook has sufficient accuracy and quality to be added to the playbook database, and that at least one playbook is subsequently added to the playbook database. The technical benefit is that only accurate, appropriate, and high-quality playbooks are identified as at least one of several playbooks and added to the playbook database. The technical benefit of validation based on similarity scores is that only distinguished playbooks related to distinguished tasks are added to the playbook database, enabling efficient use of memory and minimizing errors in matching tasks with playbook descriptors. In particular, this effect can be achieved by reducing the use of computational resources such as memory and processing power when at least one of several playbooks is added to the playbook database.

[0022] In one embodiment, the step of performing at least one validation check on multiple playbooks is at least one of the following: For each playbook in a set of playbooks stored in a playbook database, generate a number of similarity scores indicating the similarity between the corresponding descriptor of the playbook in the playbook database and the corresponding descriptors of the multiple playbooks. This includes generating a quality score indicating the quality of the corresponding set of subtasks in the plurality of playbooks, based on the plaintext descriptions of the corresponding set of subtasks in the plurality of playbooks. The step of determining whether at least one of the aforementioned playbooks passes the at least one validation check is at least one of the following: Comparing the plurality of similarity scores with a threshold similarity score, and determining that at least one of the plurality of playbooks passes the at least one validity check if the similarity score associated with a corresponding descriptor is less than the threshold similarity score. Comparing a quality score indicating the quality of a corresponding set of subtasks in the plurality of playbooks with a threshold quality score, the method includes determining that at least one of the plurality of playbooks passes the at least one validity check if the quality score associated with the corresponding set of subtasks is equal to or greater than the threshold quality score.

[0023] In this context, the term “similarity score” refers to a score indicating the degree of similarity between corresponding descriptors of any two playbooks. It will be understood that the similarity score between any two playbooks is generated using the embedding of the corresponding descriptors of these two playbooks. Optionally, various different metrics generated using the embedding of corresponding descriptors may be used as the similarity score between these two playbooks. Depending on the case, the cosine vector value generated using the corresponding descriptors of any two playbooks is preferably used as the similarity score between these two playbooks. In particular, a high similarity score indicates low distinguishability between the two playbooks, while a low similarity score indicates high distinguishability between the two playbooks. In this context, each of the multiple similarity scores is used to indicate the similarity between a playbook in the playbook database and a playbook among the multiple playbooks. Throughout this disclosure, the term “playbook in the playbook database” refers to any single playbook among the set of playbooks in the playbook database. Here, the multiple similarity scores include the similarity scores between each of the playbooks in the playbook database and the multiple playbooks. It will be understood that multiple similarity scores are generated for each playbook in the playbook database. Therefore, similarity scores are generated between each playbook in the playbook database and each playbook among the multiple playbooks.

[0024] Throughout this disclosure, the term “quality score” refers to a score that evaluates the quality of multiple playbooks by evaluating the quality of corresponding sets of subtasks. It will be understood that for each of the multiple playbooks, a separate quality score is generated for each corresponding set of subtasks. In particular, the quality score is generated by checking predetermined parameters in the plaintext descriptions of the corresponding sets of subtasks in the multiple playbooks. The predetermined parameters may be at least one of the following: whether a clear and detailed description exists for each subtask in the plaintext descriptions of the corresponding sets of subtasks in the multiple playbooks; whether the means for executing each subtask (i.e., whether at least one external tool is used or another playbook is used) are described in the plaintext descriptions; whether a logical relationship exists between each subtask in the plaintext descriptions of the corresponding sets of subtasks in the multiple playbooks; and the accuracy of each subtask in the plaintext descriptions. In other words, factors such as relevance, accuracy, feasibility, and sufficiency of the plaintext descriptions of the corresponding sets of subtasks in the multiple playbooks are evaluated to generate a quality score for the corresponding sets of subtasks in the multiple playbooks. In particular, for any playbook to be of good quality, each subtask in the corresponding set of subtasks within that playbook must have at least one reference to an external tool or another playbook for the execution of that subtask.

[0025] If any subtask in any set of corresponding subtasks in any playbook provides references to other playbooks, the quality of those other playbooks is considered good. Furthermore, if at least one external tool is provided reference to any subtask in any set of corresponding subtasks in any playbook, a detailed explanation of the parameters regarding the use of that external tool is not required, and the LLM can adequately infer and understand how to use it. Optionally, GPT-4® may be used to generate quality scores for sets of corresponding subtasks in multiple playbooks. In particular, if any subtask in any set of corresponding subtasks in any playbook provides references to other playbooks, it should be ensured that those other playbooks are different from the playbooks referenced in that subtask to avoid circular dependencies. The technical effect is that when performing at least one validation check on multiple playbooks, different factors such as the distinctiveness and quality of each playbook among the multiple playbooks are examined. Optionally, the quality score for each playbook in a set of corresponding subtasks may include a description of the quality score and suggestions on what changes should be made to the corresponding set of subtasks to improve the quality score. In particular, the description and suggestions are provided in JSON format. In this context, the term “threshold similarity score” refers to a threshold similarity score used to compare multiple similarity scores to a threshold similarity score for at least one of a set of playbooks. In particular, if the similarity score of at least one of a set of playbooks is lower than the threshold similarity score, it indicates that the at least one playbook is sufficiently distinguishable compared to a set of playbooks in the playbook database. Thus, at least one of the set of playbooks passes at least one validation check. Throughout this disclosure, the term “threshold quality score” refers to a threshold quality score used to compare quality scores to a threshold quality score.In particular, if the quality score of at least one of several playbooks is higher than the threshold quality score, it indicates that the at least one playbook is of sufficient quality to be added to the playbook database. Therefore, at least one of several playbooks passes at least one validation check. The technical effect is that at least one validation check can only be passed if at least one of several playbooks is appropriate and of good quality. In particular, this effect can be achieved by reducing the use of computational resources such as memory and processing power when determining whether at least one of several playbooks passes at least one validation check. Furthermore, the method includes obtaining a user query received by a Large Language Model (LLM), the user query relating to a request to execute a given task.

[0026] Throughout this disclosure, the term “user query” refers to a form of request that a user makes to the LLM for the execution of a given task. In particular, a user query includes information about a given task by describing or indicating the task that the user wishes to perform within the user query. Optionally, a user query is provided to the LLM by the user using a user interface on the user device. Optionally, the user device may provide the user with a text box for entering the user query. It will be understood that the LLM’s acquisition of the user query enables the LLM to process and understand the user query and determine what the given task that the user wishes to perform is. In particular, the LLM uses its ability to understand human language to determine the given task from the user query. Throughout this disclosure, the term “given task” refers to a specific computer implementation task or operation related to any technical field.

[0027] In one embodiment, the user query obtained is "Why is the website example.com not working as expected?", and this user query concerns the execution of a given task: "Determine why the website example.com is not working as expected." Furthermore, the method includes determining whether a given task matches a corresponding descriptor in any of the playbooks in a set of playbooks. In this regard, since a given task is determined from a user query, it is necessary to determine from the set of playbooks which playbook will appropriately guide the execution of that given task. In particular, for any playbook in the set of playbooks, if the corresponding descriptor of that playbook indicates that the task it guides to execute is the same as the given task, then that playbook is determined to be a given playbook which will appropriately guide the execution of the given task. It will be understood that the match between a given task and the corresponding descriptor of any playbook indicates that the task it guides to execute is the same as the given task. Subsequently, whether a given task matches the corresponding descriptor in any of the playbooks is determined by comparing the given task with the corresponding descriptor in each playbook within the set of playbooks.

[0028] Optionally, the step of determining whether a given task matches a corresponding descriptor in any of the playbooks in a set of playbooks includes comparing the context of the corresponding descriptor in the set of playbooks with the context of the given task, and if the context of the corresponding descriptor in any of the playbooks matches the context of the given task, that playbook is selected as the given playbook. Here, the term “context of the corresponding descriptor” refers to an understanding of the information conveyed by the corresponding descriptor, i.e., an understanding of what the corresponding task is that each playbook in the set of playbooks is supposed to execute. In particular, the content of each playbook in the set of playbooks is integrated into the context of the corresponding descriptor in the set of playbooks. Optionally, the context of the corresponding descriptor in the set of playbooks may include only the corresponding title of each playbook in the set of playbooks, thereby reducing the number of characters in the context of that corresponding descriptor. The term “context of the given task” refers to an understanding of what the given task is that is supposed to be executed. In particular, by comparing the context of the corresponding descriptor in a set of playbooks with the context of a given task, the understanding of which task each playbook in the set guides execution to is compared with the understanding of which given task should be executed. As a result, a match between the context of a corresponding descriptor in any playbook and the context of a given task indicates that the understanding of which task that playbook guides execution to is similar to the understanding of which given task should be executed. Therefore, that playbook is selected as the given playbook for guiding the execution of the given task. The technical effect is that it is possible to determine whether a given task matches the corresponding descriptor in any playbook without considering the entire content of any playbook in the set. In particular, this effect can be achieved by reducing the use of computational resources such as memory and processing power when comparing the context of the corresponding descriptor in a set of playbooks with the context of a given task.This allows for comparison with the context of a given task, considering only the context of the corresponding descriptors in the set of playbooks, which can be achieved with less processing power and memory.

[0029] In one embodiment, LLM is used in the step of determining whether a given task matches a corresponding descriptor in any of the playbooks in a set of playbooks, the step of comparing the context of the corresponding descriptor in the set of playbooks with the context of the given task.

[0030] Optionally, the step of determining whether a given task matches a corresponding descriptor in any of the playbooks in the set of playbooks is further: This involves generating embeddings for the corresponding descriptors of a set of playbooks and storing those embeddings in a vector database. Converting user queries into query embeddings, This includes identifying a playbook as a given playbook that has a corresponding embedding that most closely resembles the query embedding, based on at least one of a cosine similarity score and a threshold.

[0031] In this context, the term “embedding” refers to a vector representation of the corresponding descriptors in a set of playbooks. As a technical effect, generating embeddings of the corresponding descriptors in a set of playbooks can favorably reduce the amount of memory required to store information about the corresponding descriptors in a set of playbooks. Throughout this disclosure, the term “vector database” refers to a database created for the purpose of storing vector values. Therefore, the generated embeddings of the corresponding descriptors in a set of playbooks are stored in the vector database. Throughout this disclosure, the term “query embedding” refers to a vector representation of a user query. In particular, since embeddings of the corresponding descriptors in a set of playbooks have been generated, the user query is converted into a query embedding so that the user query can be matched with the corresponding descriptors in a set of playbooks. The corresponding embeddings in the set of playbooks and the query embedding are then matched to determine whether a given task matches the corresponding descriptor in any of the playbooks in the set of playbooks.

[0032] In this regard, for each playbook in a set of playbooks, a similarity score is generated using the playbook's embeddings and query embeddings to indicate the similarity between the playbook and the user query. In particular, various types of similarity scores may be generated to indicate the similarity between corresponding embeddings and query embeddings in a set of playbooks. A similarity score close to 0 between a particular playbook and a user query indicates that the playbook and the user query are significantly different, while a score close to 1 indicates that the playbook and the user query are highly similar. A cosine similarity score is a type of similarity score generated between corresponding embeddings and query embeddings in a set of playbooks, and it will be understood that the cosine similarity score indicates the similarity between the two. In the implementation, the playbook with the highest cosine similarity score among all the cosine similarity scores generated for the set of playbooks is identified as the playbook whose corresponding embeddings most closely resemble the query embeddings. The similarity score of the most similar playbook is compared to a predetermined threshold (preferably 0.75) in the range of 0 to 1 to determine whether the similarity between the corresponding descriptor of the playbook and the user query is at a sufficient level. Subsequently, for the similarity between the corresponding descriptor of the playbook and the user query to be at a sufficient level, the similarity score of the closest playbook must be greater than the threshold. Optionally, this threshold may be set through a trial-and-error process. If the threshold is too low, an inaccurate playbook may be selected as the given playbook, while if the threshold is too high, an accurate playbook may not be selected as the given playbook.

[0033] For example, vector database "D" contains embeddings of corresponding descriptors generated for a set of playbooks. A query embedding "q" is generated for a user query "Q". Next, query embedding "q" is compared with all embeddings stored in vector database "D" to identify playbook "A" which has the corresponding embedding "v" that most closely resembles query embedding "q". Subsequently, the similarity score between playbook "A" and the user query is compared to a threshold of 0.75 to determine whether the similarity between the corresponding descriptor in playbook "A" and a given task in the user query is at a sufficient level. If the similarity score between playbook "A" and the user query is greater than 0.75, playbook "A" is selected as the given playbook to guide the execution of the given task.

[0034] As a technical benefit, generating an embedding of the corresponding descriptor in a set of playbooks can advantageously reduce the amount of memory required to store the information of the corresponding descriptor. In particular, this effect can be achieved by reducing the use of computational resources (memory and processing power, etc.) when generating the embedding of the corresponding descriptor in a set of playbooks. Furthermore, if it is determined that a given task matches a given descriptor in a given playbook within a set of playbooks, the LLM selects one or more external tools, and these one or more external tools read the plaintext description of a given set of subtasks. Throughout this disclosure, the term “given playbook” refers to a specific playbook within a set of playbooks that is suitable for appropriately guiding the execution of a given task because the task it guides the execution of is identical to that of a given task. Throughout this disclosure, the term “given descriptor” refers to a corresponding descriptor in a given playbook that indicates what task the given playbook guides the execution of. In this regard, determining that a given task matches a given descriptor in a given playbook within a set of playbooks indicates that a suitable playbook has been found within the set of playbooks to appropriately guide the execution of the given task. The given playbook is then used to guide the execution of the given task according to the steps of the method described above.

[0035] In particular, by guiding the execution of a given task using a given playbook, it is ensured that the given task is always executed with accurate and consistent results. Throughout this disclosure, the term “one or more external tools” refers to one or more software applications that are accessible and available to perform a particular computational function or operation. In particular, each of the one or more external tools is used to perform a particular computational operation. Examples of one or more external tools include, but are not limited to, dig, curl, nmap, whois, ping, opens, visualize-IP, etc. In this regard, one or more external tools are selected by the LLM based on their suitability and ability to perform a given task. In particular, one or more external tools are used to read a plaintext description of a given set of subtasks in order to understand what the necessary steps or instructions to follow for the execution of a given task are. It will be understood that the understanding of the plaintext description of a given set of subtasks formed by one or more external tools is then sent to the software framework.

[0036] Optionally, the suitability and capability of one or more external tools for performing a given task is determined by the LLM using factors such as the external tool descriptors, the data returned by the external tools, and external tools enumerated in relation to a given set of subtasks in a given playbook. Throughout this disclosure, the term “descriptor of multiple external tools” refers to attributes or characteristics of multiple external tools that describe what computational operations each external tool performs. Throughout this disclosure, the term “data returned by multiple external tools” refers to information about the type of output data obtained by using each of the multiple external tools. Throughout this disclosure, the term “external tools enumerated in relation to a given set of subtasks in a given playbook” refers to external tools described in the plaintext description of a given set of subtasks in a given playbook and used for performing that set of subtasks. In this regard, the descriptors of multiple external tools, the data returned by multiple external tools, and the external tools listed in relation to a given set of subtasks in a given playbook are multiple factors used to determine which of the multiple external tools is suitable for performing a given task, and at least one of these factors is used.

[0037] Optionally, the method further includes generating a first response to a user query and sending the first response via LLM to a user device, the first response including a plaintext description of a given set of subtasks in a given playbook. Throughout this disclosure, the term “first response” refers to a computer-generated response generated in response to a user query to inform the user of what processing is planned for the execution of a task. In particular, the first response is generated by the LLM in human natural language. Throughout this disclosure, the term “user device” refers to a computing device associated with a user, the user device is capable of communicating with the LLM and receiving the first response from the LLM. Optionally, the user device may be a computer, laptop, mobile phone, tablet, etc.

[0038] In one embodiment, the first response sent to the user device is as follows: "Understood. We will diagnose why the website 'example.com' is not working as expected. To identify the problem, please follow the steps below." 1. Verify DNS configuration: Confirm that the DNS server is correctly configured for the domain and is returning a valid IP address. 2. Check port forwarding status: Verify that ports 80 and 443 are open on the IP address. 3. Verify HTML and SSL configuration: If both ports are open, use curl to verify that the ports are returning valid HTML pages and that there are no problems with the HTTPS SSL configuration. 4. Verify single port opening: If only one port is open, verify that valid HTML is being returned. 5. Check if ports are open: If none of the web ports are open, this could be the problem. 6. Final Check: If everything appears normal, perform a final check from each continental location to ensure there are no routing issues and that all responses are similar. This is the case when the user query is "Why isn't the website example.com working as expected?"

[0039] Furthermore, a software framework associated with an LLM works with the LLM to track and execute a given set of subtasks in a given playbook using one or more external tools. Throughout this disclosure, the term “software framework” refers to a type of computing software that is integrable with an LLM and enhances the functionality and capabilities of the LLM. Optionally, a software framework may include one or more software applications, each software application within the software framework assigned to perform a specific function. In particular, executing a given set of subtasks in a given playbook requires that each subtask in that set be executed in a predetermined order. In this regard, the execution of each subtask in that set is performed by using one or more external tools to perform the computational operations corresponding to each subtask. In other words, one or more external tools are accessed and used to perform the computational operations corresponding to each subtask in that set. It will be understood that the software framework tracks and executes a given set of subtasks in a given playbook by receiving an understanding of the plaintext description of a given set of subtasks formed by one or more external tools. In particular, LLM is used to coordinate the execution of each individual subtask within a given set of subtasks, thereby enabling consistent and accurate results from the execution of a given set of subtasks in a given playbook.

[0040] Furthermore, the LLM generates a query response containing the results of executing a given set of subtasks and sends the query response to the user device. Throughout this disclosure, the term “query response” refers to the final response created to provide the results of a user query. It will be understood that the query response indicates whether the retrieved user query was processed correctly. In this regard, since the user query concerns a request to execute a given task, the query response includes the results of executing a given set of subtasks in order to provide the results of the user query. Optionally, the results of executing a given set of subtasks may be in the form of numerical data or at least one of a description indicating the success or failure of the execution of a given task. In particular, the LLM is used to mimic human interaction with the user in the generated query response. It will be understood that the LLM is communicateably connected to the user device in order to send the query response to the user device. Optionally, the query response is provided to the user in the graphical user interface of the user device.

[0041] Optionally, if it is determined that a given task does not match a corresponding descriptor in any of the playbooks in the set of playbooks, this method further includes creating a new playbook and adding it to the playbook database, namely, The new playbook receives a corresponding descriptor indicating the given task and a plaintext description of a corresponding set of subtasks to be performed for the execution of the given task. Perform at least one validation check on the aforementioned new playbook. To determine whether the new playbook passes at least one of the validation checks, If the new playbook is determined to pass at least one of the validity checks, Adding the aforementioned new playbook to the set of playbooks in the playbook database, The process includes repeating the step of determining whether the given task matches a corresponding descriptor in any of the playbooks in the set of playbooks.

[0042] In this context, the term “new playbook” refers to a playbook created separately to guide the execution of a given task when it is determined that the given task does not match the corresponding descriptor in any of the playbooks in a set of playbooks. It will be understood that new playbooks are created and added to the playbook database directly by the user via at least one software framework, or by a technical expert in the technical field related to the given task. In particular, the information and criteria required to add a new playbook to the playbook database are the same as those used when adding a set of playbooks to the playbook database. The plaintext descriptions of the corresponding descriptors and corresponding sets of subtasks for the new playbook are received from the user via the user device. It will be understood that receiving the plaintext descriptions of the corresponding descriptors and corresponding sets of subtasks for the new playbook ensures that sufficient information is available to add the new playbook to the playbook database. In particular, the corresponding descriptor of the new playbook indicates the given task, thus ensuring that the given task matches the corresponding descriptor of the new playbook, and therefore the new playbook is selected as the given playbook to guide the execution of the given task. Furthermore, a step is performed to verify the accuracy and quality of the new playbook by performing at least one validation check on the new playbook. The new playbook is added to the set of playbooks in the playbook database only if it is determined that the new playbook passes at least one validation check. After the new playbook is added to the playbook database, the step of determining whether a given task matches a corresponding descriptor in any of the playbooks in the set is repeated, and as a result, the new playbook is selected as the given playbook to guide the execution of the given task.

[0043] As a technical benefit, it becomes possible to easily add appropriate user-defined playbooks to the playbook database as new playbooks to guide the execution of a given task. In particular, this effect can be achieved by reducing the use of computing resources such as memory and processing power when adding new playbooks to the playbook database.

[0044] Optionally, if it is determined that a given task does not match the corresponding descriptor in any of the playbooks in the set of playbooks, this method further: The process involves generating an action plan for a given task using LLM, wherein the action plan includes a set of LLM-generated subtasks to be performed for the execution of the given task. Using descriptors for multiple external tools, data returned by multiple external tools, and at least one of the external tools enumerated in relation to the set of LLM-generated subtasks in the action plan, LLM selects one or more external tools from among the multiple external tools to perform a given task. Using one or more of the aforementioned external tools, execute the set of LLM generation subtasks in the action plan. The process includes generating a query response to a user query using an LLM and sending the query response from the LLM to the user device, wherein the query response includes at least one of the data obtained by executing a set of LLM generation subtasks by one or more external tools and the data obtained by executing the last LLM generation subtask of the set of LLM generation subtasks.

[0045] In this context, the term “Action Plan” refers to a description of an auxiliary plan generated by the LLM when no suitable or appropriate playbook exists in the playbook database to guide the execution of a given task. The LLM is used as a task planning agent to generate the action plan. In particular, the action plan includes information about the steps to be performed for the execution of a given task. It will be understood that this information about the steps is included in the action plan in the form of a set of LLM-generated subtasks. Throughout this disclosure, the term “Set of LLM-generated subtasks” refers to one or more intermediate tasks related to the steps to be performed for the execution of a given task, which are generated using the reasoning and logical capabilities of the LLM. Subsequently, one or more external tools are selected from among several external tools using one or more factors that determine the external tools necessary to execute the set of LLM-generated subtasks. In this context, one or more external tools are accessed and used to perform the operations and functions necessary to execute the set of LLM-generated subtasks. In particular, the results generated after executing a set of LLM generation subtasks by one or more external tools are collected in the form of at least one of the following: a summary of the data returned by one or more external tools as a result of executing the set of LLM generation subtasks in the action plan, and the data returned as a result of executing the last LLM generation subtask in the set of LLM generation subtasks in the action plan. Throughout this disclosure, the term “query response” refers to a response that contains information about the final results of executing a given task under the guidance of the action plan. As a technical effect, it becomes possible to execute a given task even if a suitable playbook for guiding the execution of the given task does not exist in the playbook database.

[0046] In one embodiment, the LLM is trained to generate action plans by being fine-tuned using a supervised learning algorithm or a reinforcement learning technique with human feedback. Examples of LLMs trained through fine-tuning include BloombergGPT® and SecureFalcon®.

[0047] In another embodiment, the LLM is trained to generate action plans by employing various prompting techniques to improve the LLM's reasoning ability. For example, few-shot prompting enables in-context learning of the LLM by providing in-domain knowledge or pairs of input and output examples using natural language. Other prompting techniques include Chain-of-thought (CoT) prompting, Plan-and-solve prompting, and zero-shot CoT prompting, which guide the LLM to construct a set of subtasks in its action plan.

[0048] Optionally, in any playbook within a set of playbooks, the plaintext description of a given subtask within a corresponding set of subtasks in that playbook includes a reference to one or more other playbooks within the set of playbooks in the playbook database, and when that given subtask is executed, the corresponding set of subtasks in one or more other playbooks is executed. In this specification, the term “given subtask” refers to any particular subtask within a set of corresponding subtasks in a playbook. In particular, the provision of a reference to one or more other playbooks within a given subtask indicates that the execution of that given subtask should be guided by the one or more other playbooks that are referenced. It will be understood that one or more other playbooks are referenced by including the corresponding descriptor of that one or more other playbooks in the plaintext description of the given subtask. For example, in the plaintext description of a given subtask, "Use the playbook DNS-MEASURE to evaluate the DNS security status of a domain. This step must always be performed," a reference to another playbook is made by including the corresponding descriptor "DNS-MEASURE." The set of corresponding subtasks includes one or more sets of corresponding subtasks from one or more other playbooks referenced in the given subtask. In this regard, one or more other playbooks are used to induce the execution of the given subtask only by executing the set of corresponding subtasks in the one or more other playbooks referenced in the given subtask. The technical effect is that the method of execution of the given subtask is clearly defined in the plaintext description of the given subtask. In particular, since the execution of the given subtask only requires following the fixed instructions provided in the plaintext description, this effect can be achieved by reducing the use of computing resources such as processing power.

[0049] Optionally, one or more external tools are accessed through one or more application programming interfaces (APIs). In this specification, “application programming interface” means a set of rules or protocols that enable interaction and communication with one or more external tools. In particular, each of the one or more external tools is accessed through a corresponding API among the one or more APIs. It will be understood that one or more APIs function as entry points for accessing one or more external tools to perform the necessary computational operations described in the plaintext description of a given set of subtasks in a given playbook. The technical effect is that one or more external tools become easily accessible and usable through one or more APIs. In particular, since one or more APIs function merely as a means of communication for interacting with one or more external tools and do not require significant processing, this effect can be achieved by reducing the use of computational resources such as processing power.

[0050] Optionally, a given set of subtasks in a given playbook is executed in at least one of sequential, graphical, or dynamically changing order. In this regard, a predetermined method by which a given set of subtasks should be executed in order to appropriately guide the execution of a given task by a given playbook is selected as at least one of sequential, graphical, or dynamically changing order. In particular, executing a given set of subtasks in a given playbook in at least one of sequential, graphical, or dynamically changing order makes it possible to determine how any subtask in a given set of subtasks relates to other subtasks. In other words, by executing a given set of subtasks in sequential or graphical order, conditional relationships are established between subtasks in that set. Once such conditional relationships are established, the execution of a given subtask becomes dependent on the outcome of at least one other subtask in that set, and as a result, the original sequential or graphical order may be changed. Therefore, the execution order of a given set of subtasks in a given playbook can be dynamically changed to enable more efficient execution. This improvement in efficiency directly contributes to a reduction in the computational resources required to execute a given set of subtasks in a given playbook.

[0051] It will be understood that by executing a given set of subtasks in a given playbook in at least one of sequential, graphical, or dynamically changing order, accurate and consistent results will always be produced when a given task is executed under the guidance of the given playbook. A technical benefit is that a given playbook conveniently and clearly defines when and in what order each subtask in a given set of subtasks should be executed. In particular, this benefit can be achieved by reducing the use of computational resources, such as processing power, when executing a given set of subtasks in a given playbook in at least one of sequential, graphical, or dynamically changing order.

[0052] Optionally, a given task may be a cybersecurity task. In this context, the term “cybersecurity task” refers to a computational task related to identifying vulnerabilities in software applications that could be exploited for malicious purposes. Furthermore, such cybersecurity tasks typically involve preventing access to computer systems by undesirable entities. In particular, cybersecurity tasks require the execution of complex, multi-step logical operations to obtain accurate and consistent results each time they are performed. Therefore, the benefit of the method when a given task is a cybersecurity task lies in its ability to accurately and consistently perform highly complex tasks related to web security analysis and diagnosis. In particular, this benefit can be achieved by reducing the use of computational resources such as memory and processing power when performing a cybersecurity task by executing a given set of subtasks in a given playbook, where the playbook includes a description of the corresponding set of subtasks to be executed for the task. This makes it possible to accurately and completely perform even complex tasks that require multi-step analysis and typically require relatively large processing power and / or computational memory with significantly less computational power. By using the system of the present invention to carry out the method of the present invention, consistent responses are guaranteed even for complex tasks, thus eliminating the need to rerun tasks and resulting in the saving of computational resources.

[0053] This disclosure also relates to the system described above. The various embodiments and modifications disclosed with respect to the aforementioned method are also applicable to that system with necessary modifications.

[0054] Throughout this disclosure, the term “data repository” refers to a storage medium configured to store a playbook database. Optionally, a data repository may include, but is not limited to, internal storage devices, external storage devices, Universal Serial Bus (USB), hard disk drives (HDDs), flash memory, Secure Digital (SD) cards, solid-state drives (SSDs), computer-readable storage media, GitHub® repositories, or any appropriate combination thereof.

[0055] Throughout this disclosure, the term “processor” refers to a computing element capable of executing a software framework. Examples of processors include, but are not limited to, microprocessors, microcontrollers, complex instruction-set computing (CISC) microprocessors, reduced instruction-set (RISC) microprocessors, very long instruction word (VLIW) microprocessors, or any other type of processing circuit. Furthermore, a processor may refer to one or more individual processors, processing units, and various elements related to processing units that may be shared with other processing units. It will also be understood that one or more individual processors, processing units, and elements may be configured in various architectures to process instructions in response to instructions executing a software framework. The term “at least one processor” refers to “one processor” in some embodiments and to “multiple processors” in other embodiments.

[0056] In particular, by having at least one processor communicatively coupled to the data repository, the software framework can access the playbook database as needed. Optionally, to create a playbook database, the software framework further... For multiple playbooks, receive plaintext descriptions of the corresponding descriptors and corresponding sets of subtasks. Perform at least one validation check on the aforementioned multiple playbooks, Determine whether at least one of the aforementioned playbooks passes the at least one validation check. The system is configured to add at least one of the aforementioned playbooks to the playbook database if it is determined that at least one of the aforementioned playbooks passes the at least one validation check.

[0057] Optionally, to perform at least one validation check against multiple playbooks, the software framework may further include at least one of the following: For each playbook in the set of playbooks stored in the playbook database (302), generate a plurality of similarity scores that show the similarity between the corresponding descriptor of the playbook and the corresponding descriptors of the plurality of playbooks. Based on the plaintext descriptions of the corresponding sets of subtasks in the multiple playbooks, the system generates a quality score indicating the quality of the corresponding sets of subtasks in the multiple playbooks. Furthermore, in order to determine whether at least one of the plurality of playbooks passes the at least one validity check, at least one of the following, namely, Comparing the plurality of similarity scores with a threshold similarity score, and determining that at least one of the plurality of playbooks passes the at least one validity check if the similarity score associated with a corresponding descriptor is less than the threshold similarity score. The process involves comparing the quality score indicating the quality of corresponding sets of subtasks in the plurality of playbooks with a threshold quality score, and determining that at least one of the plurality of playbooks passes the at least one validity check if the quality score associated with the corresponding set of subtasks is equal to or greater than the threshold quality score.

[0058] Optionally, in order to determine whether a given task matches a corresponding descriptor in any of the playbooks in the set of playbooks, the software framework is further configured to compare the context of the corresponding descriptor in the set of playbooks with the context of the given task, and if the context of the corresponding descriptor in any of the playbooks matches the context of the given task, that playbook is selected as the given playbook (314, 400).

[0059] Optionally, in order to determine whether a given task matches the corresponding descriptor of any of the playbooks in a set of playbooks, the software framework further: Generate embeddings for the corresponding descriptors of the playbook set, and store the embeddings in a vector database. Convert user query (308) into query embedding, Based on at least one of the cosine similarity score and a threshold, the playbook having the corresponding embedding by most closely resembling the query embedding is identified as a given playbook (314, 400).

[0060] Optionally, if it is determined that a given task does not match the corresponding descriptor of any of the playbooks in the set of playbooks, the software framework further creates a new playbook to be added to the playbook database (302). With respect to the new playbook, receive a corresponding descriptor indicating the given task and a plaintext description of a set of corresponding subtasks to be performed for the execution of the given task. Perform at least one validation check on the aforementioned new playbook, Determine whether the new playbook passes the at least one validation check. If the new playbook is determined to pass at least one of the validity checks, The new playbook is added to the set of playbooks in the playbook database. The system is configured to repeat the step of determining whether the given task matches a corresponding descriptor in any of the playbooks in the set of playbooks.

[0061] Optionally, in any playbook in a set of playbooks, the plaintext description of a given subtask in a corresponding set of subtasks in that playbook includes a reference to one or more other playbooks in the playbook database (302), and when the given subtask is executed, the corresponding set of subtasks in the one or more other playbooks is executed.

[0062] Optionally, the set of subtasks in a given playbook (314, 400) is executed in at least one of the following order: sequential, graphical, or dynamically changing. Optionally, a given task is a cybersecurity task.

[0063] This disclosure also relates to the computer program product described above. The various embodiments and modifications disclosed with respect to the aforementioned methods and systems are also applicable to the computer program product with necessary modifications.

[0064] Tests were conducted to evaluate the robustness of the playbook selection mechanism of this disclosure. In other words, tests were conducted to verify whether the same playbook is selected (i.e., determined to be a match for a given task) when different prompts (i.e., different user queries or different variations of a user query) with similar meanings (i.e., related to a given task) are obtained.

[0065] In the test, one user query was obtained, and 40 variations of it (i.e., 40 similar user queries with different degrees of similarity) were generated using LLM. For example, one user query was "Assess the security status of example.com." In this context, some of the 40 variations of this user query were "Conduct a security assessment of example.com," "Audit example.com from a security perspective," "Are there any security warnings for example.com?", "What are some potential issues that could compromise the security of example.com?", and "Is example.com adequately protected against online attackers?". Subsequently, for each variation of the user query, it was checked whether the task corresponding to that variation matched the corresponding descriptor in one of the playbooks in the set of playbooks. The test results showed that the same playbook was selected each time (i.e., the same playbook matched each (different) variation of the user query). This 100% accuracy demonstrates the robustness of the playbook selection mechanism of this disclosure and also shows that the innovative playbook-based approach of this disclosure is unbiased towards the titles (i.e., descriptors) provided in the set of playbooks.

[0066] Referring together to Figures 1A and 1B, a flowchart 100 is shown illustrating the steps of a method for inducing large language models (LLMs) for task planning and execution according to one embodiment of the present disclosure. Step 102 creates a playbook database containing a set of playbooks, each playbook having a corresponding descriptor indicating a task to be induced to be executed and containing a plaintext description of a corresponding set of subtasks to be executed for that task. Step 104 retrieves a user query received by a large language model (LLM) relating to a request to execute a given task. Step 106 determines whether a given task matches a corresponding descriptor in any of the playbooks in the set of playbooks. If it is determined that a given task matches a given descriptor in a given playbook in the set of playbooks, then in step 108, the LLM selects one or more external tools, which read the plaintext descriptions of a given set of subtasks. In step 110, the software framework associated with the LLM works with the LLM to track and execute a given set of subtasks in a given playbook using one or more of the aforementioned external tools. In step 112, the LLM generates a query response containing the execution results of the given set of subtasks and sends the query response to the user device. Optionally, if it is determined that a given task does not match a corresponding descriptor in any of the playbooks in the set of playbooks, in step 114, a new playbook is created to be added to the playbook database.

[0067] Referring to Figure 2, a block diagram of a system 200 for guiding large-scale language models (LLMs) for task planning and execution according to one embodiment of the present disclosure is shown. As shown, the system 200 comprises a data repository 202. Furthermore, the system 200 comprises at least one processor (indicated as processor 204), the at least one processor 204 being communicatively coupled to the data repository 202. The at least one processor 204 is configured to run a software framework 206 to create a playbook database in the data repository 202 containing a set of playbooks, each playbook having a corresponding descriptor indicating a task whose execution is to be guided and containing a plaintext description of a corresponding set of subtasks to be executed for the execution of that task. Furthermore, the software framework 206 is executed to retrieve user queries received by a large-scale language model (LLM) 208, the user queries relating to requests for execution of a given task. Furthermore, the software framework 206 is executed to determine whether a given task matches a corresponding descriptor in any of the playbooks in the set of playbooks. Furthermore, if it is determined that a given task matches a given descriptor in a given playbook within a set of playbooks, the LLM208 selects one or more external tools 212, which read the plaintext description of the given set of subtasks. Furthermore, the software framework 206 associated with the LLM works with the LLM208 to execute the given set of subtasks in the given playbook using one or more external tools 212. Furthermore, the LLM208 generates a query response containing the execution results of the given set of subtasks and sends the query response to the user device 210.

[0068] Referring to Figure 3, a flowchart is shown illustrating an implementation scenario of a system 300 for guiding large-scale language models (LLMs) for task planning and execution, according to one embodiment of the present disclosure. As shown, a playbook database 302 containing a set of playbooks is created using a playbook submission tool 304. Furthermore, a language model (LLM) 306 receives a user query 308, which concerns a request to execute a given task. Subsequently, a software framework 310 retrieves the user query 308 received by the LLM 306. The software framework 310 then uses a playbook selection tool 312 to determine whether a given task matches a corresponding descriptor in one of the playbooks in the set of playbooks in the playbook database 302. Furthermore, if it is determined that a given task matches a given descriptor in a given playbook 314, the given set of subtasks in the given playbook 314 is executed using one or more external tools (indicated as external tool 316), and the data obtained from the execution of the given set of subtasks in the given playbook 314 using one or more external tools 316 is used to generate a second response 318 for the user. On the other hand, if it is determined that a given task does not match a corresponding descriptor in any of the playbooks in the set of playbooks in the playbook database 302, the LLM generates an action plan 320 to guide the execution of the given task. Subsequently, the LLM-generated set of subtasks in the action plan 320 is executed using one or more external tools 316, and a fourth response 322 for the user is generated.

[0069] Referring to Figure 4, a flowchart is shown illustrating the graphical sequence of execution of a given set of subtasks in a given playbook 400 according to one embodiment of the present disclosure. As shown in the figure, the execution of a given set of subtasks begins with the execution of the first subtask 402 of the set of subtasks. Subsequently, the second subtask 404 of the set of subtasks is executed. If the result of the execution of the second subtask 404 is "yes", then the third subtask 406 of the set of subtasks is executed after the execution of the second subtask 404. Subsequently, the fourth subtask 408 of the set of subtasks is executed after the execution of the third subtask 406. On the other hand, if the result of the execution of the second subtask 404 is "no", then the fourth subtask 408 of the set of subtasks is executed after the execution of the second subtask 404. If the result of executing the fourth subtask 408 is "yes", then after the execution of the fourth subtask 408, the fifth subtask 410 of that set of subtasks is executed, and the result of executing the fifth subtask 410 becomes the final result of executing the given set of subtasks in the given playbook 400. On the other hand, if the result of executing the fourth subtask 408 is "no", then the result of executing the fourth subtask 408 becomes the final result of executing the given set of subtasks in the given playbook 400.

Claims

1. A method for guiding the task planning and execution of large-scale language models (LLMs), Create a playbook database (302) containing a set of playbooks, each playbook having a corresponding descriptor indicating a task that guides its execution, and containing a plaintext description corresponding to a set of subtasks to be performed for the execution of that task. The Large-Scale Language Model (LLM) (306, 208) receives a user query (308), which concerns a request to execute a given task. The given task determines whether it matches the corresponding descriptor of any of the playbooks in the set of playbooks. If it is determined that the given task matches the corresponding descriptor of a given playbook (314, 400) in the set of playbooks, The LLM selects one or more external tools (212, 316), and the one or more external tools read the plaintext descriptions of a given set of subtasks. The software framework associated with the LLM, in cooperation with the LLM, uses one or more external tools to track and execute the given set of subtasks in the given playbook. The LLM includes generating a query response containing the results of executing the given set of subtasks and sending the query response to the user device. method.

2. The step of creating the aforementioned playbook database (302) is, For multiple playbooks, receive plaintext descriptions of the corresponding descriptors and corresponding sets of subtasks. Perform at least one validation check on the aforementioned multiple playbooks, Determine whether at least one of the aforementioned playbooks passes the at least one validation check. The method according to claim 1, further comprising adding at least one of the plurality of playbooks to the playbook database if it is determined that at least one of the plurality of playbooks passes the at least one validity check.

3. The step of performing the at least one validation check on the plurality of playbooks is at least one of the following: For each playbook in the set of playbooks stored in the playbook database (302), a plurality of similarity scores are generated that indicate the similarity between the corresponding descriptor of the playbook in the playbook database and the corresponding descriptors of the plurality of playbooks. This includes generating a quality score indicating the quality of the corresponding set of subtasks in the plurality of playbooks, based on the plaintext descriptions of the corresponding set of subtasks in the plurality of playbooks. The step of determining whether at least one of the aforementioned playbooks passes the at least one validation check is at least one of the following: The plurality of similarity scores are compared with a threshold similarity score, and if the similarity score associated with the corresponding descriptor is less than the threshold similarity score, it is determined that at least one of the plurality of playbooks passes the at least one validity check. The method according to claim 2, comprising comparing the quality score indicating the quality of corresponding sets of subtasks in the plurality of playbooks with a threshold quality score, and determining that at least one of the plurality of playbooks passes the at least one validity check if the quality score associated with the corresponding set of subtasks is equal to or greater than the threshold quality score.

4. The step of determining whether the given task matches a corresponding descriptor in any of the playbooks in the set of playbooks is: This includes comparing the context of the corresponding descriptor in the set of playbooks with the context of the given task, The method according to any one of claims 1 to 3, wherein a playbook is selected as the given playbook (314, 400) if the context of a corresponding descriptor in any of the playbooks in the set of playbooks matches the context of the given task.

5. The step of determining whether the given task matches a corresponding descriptor in any of the playbooks in the set of playbooks is further: Generate embeddings for the corresponding descriptors of the set of playbooks, and store the embeddings in a vector database. The aforementioned user query (308) is converted into a query embedding. The method according to any one of claims 1 to 4, comprising identifying a playbook having a corresponding embedding that most closely approximates the query embedding as the given playbook (314, 400) based on at least one of a cosine similarity score and a threshold.

6. If it is determined that the given task does not match the corresponding descriptor of any of the playbooks in the set of playbooks, the further step is to create a new playbook to be added to the playbook database (302), i.e., With respect to the new playbook, receive a corresponding descriptor indicating the given task and a plaintext description of a set of corresponding subtasks to be performed for the execution of the given task. Perform at least one validation check on the aforementioned new playbook. Determine whether the new playbook passes the at least one validation check. If the new playbook is determined to pass at least one of the validity checks, The new playbook is added to the set of playbooks in the playbook database. The method according to any one of claims 1 to 5, comprising repeating the step of determining whether the given task matches a corresponding descriptor in any of the playbooks in the set of playbooks.

7. In any of the playbooks in the set of playbooks, the plaintext description of a given subtask in the corresponding set of subtasks in the playbook includes a reference to one or more other playbooks in the set of playbooks in the playbook database (302). The method according to any one of claims 1 to 6, wherein when the given subtask is executed, a corresponding set of subtasks in one or more other playbooks is executed.

8. The method according to any one of claims 1 to 7, wherein the one or more external tools (212, 316) are accessed via one or more application programming interfaces (APIs).

9. The method according to any one of claims 1 to 8, wherein the set of subtasks in the given playbook (314, 400) is executed in at least one of sequential order, graphical order, or dynamically changing order.

10. The method according to any one of claims 1 to 9, wherein the given task is a cybersecurity task.

11. A system (200, 300) for guiding the task planning and execution of large-scale language models (LLMs), Data repository (202), At least one processor (204) communicatively coupled to the data repository, the at least one processor is configured to execute a software framework (206, 310), the software framework is In the aforementioned data repository, a playbook database (302) is created which includes a set of playbooks, each playbook having a corresponding descriptor indicating a task that guides its execution, and including a plaintext description corresponding to a set of subtasks to be executed for the execution of that task. The Large-Scale Language Model (LLM) (306, 208) retrieves a user query (308) received, the user query relating to a request to perform a given task, Determine whether the given task matches the corresponding descriptor of any of the playbooks in the set of playbooks. If it is determined that the given task matches a given descriptor in a given playbook (314, 400) from the set of playbooks, The LLM selects one or more external tools (212, 316), and the one or more external tools read the plaintext description of the given set of subtasks. The software framework associated with the LLM, in cooperation with the LLM, uses one or more external tools to track and execute a given set of subtasks in a given playbook. The LLM generates a query response containing the results of executing the given set of subtasks and sends the query response to the user device. system.

12. To create the aforementioned playbook database (302), the software framework (206, 310) further: For multiple playbooks, receive plaintext descriptions of the corresponding descriptors and corresponding sets of subtasks. Perform at least one validation check on the aforementioned multiple playbooks, Determine whether at least one of the aforementioned playbooks passes the at least one validation check. The system according to claim 11, wherein if it is determined that at least one of the plurality of playbooks passes the at least one validity check, the at least one of the plurality of playbooks is added to the playbook database.

13. In order to perform the above-mentioned validation check on the plurality of playbooks, the software framework (206, 310) further includes at least one of the following: For each playbook in the set of playbooks stored in the playbook database (302), a plurality of similarity scores are generated that indicate the similarity between the corresponding descriptor of the playbook in the playbook database and the corresponding descriptors of the plurality of playbooks. The system is configured to generate a quality score indicating the quality of the corresponding set of subtasks in the multiple playbooks, based on the plaintext descriptions of the corresponding sets of subtasks in the multiple playbooks. In order to determine whether at least one of the plurality of playbooks passes the at least one validation check, the software framework further determines at least one of the following: The plurality of similarity scores are compared with a threshold similarity score, and if the similarity score associated with the corresponding descriptor is less than the threshold similarity score, it is determined that at least one of the plurality of playbooks passes the at least one validity check. The system according to claim 12, wherein the quality score indicating the quality of corresponding sets of subtasks in the plurality of playbooks is compared with a threshold quality score, and if the quality score associated with the corresponding set of subtasks is equal to or greater than the threshold quality score, it is determined that at least one of the plurality of playbooks passes the at least one validity check.

14. The system according to any one of claims 11 to 13, wherein the software framework is further configured to compare the context of the corresponding descriptor in the set of playbooks with the context of the given task in order to determine whether the given task matches a corresponding descriptor in any of the playbooks in the set of playbooks, and if the context of the corresponding descriptor in any of the playbooks in the set of playbooks matches the context of the given task, the playbook is selected as the given playbook (314, 400).

15. In order to determine whether the given task matches a corresponding descriptor in any of the playbooks in the set of playbooks, the software framework (206, 310) further: Generate embeddings for the corresponding descriptors of the set of playbooks, and store the embeddings in a vector database. The aforementioned user query (308) is converted into a query embedding. The system according to any one of claims 11 to 14, configured to identify a playbook having a corresponding embedding that best approximates the query embedding based on at least one of a cosine similarity score and a threshold as the given playbook (314, 400).

16. If it is determined that the given task does not match the corresponding descriptor of any of the playbooks in the set of playbooks, the software framework (206, 310) further creates a new playbook to be added to the playbook database (302), With respect to the new playbook, receive a corresponding descriptor indicating the given task and a plaintext description of a set of corresponding subtasks to be performed for the execution of the given task. Perform the aforementioned new playbook and at least one validation check on it. Determine whether the new playbook passes the at least one validation check. If the new playbook is determined to pass at least one of the validity checks, The new playbook is added to the set of playbooks in the playbook database. The system according to any one of claims 11 to 15, configured to repeat the step of determining whether a given task matches a corresponding descriptor in any of the playbooks in the set of playbooks.

17. The system according to any one of claims 11 to 16, wherein in any playbook of the set of playbooks, the plaintext description of a given subtask of a corresponding set of subtasks in the playbook includes a reference to one or more other playbooks of the set of playbooks in the playbook database (302), and when the given subtask is executed, the corresponding set of subtasks in the one or more other playbooks is executed.

18. The system according to any one of claims 11 to 17, wherein the set of subtasks in the given playbook (314, 400) is configured to be executed in at least one of sequential, graphical, or dynamically changing order.

19. The system according to any one of claims 11 to 18, wherein the given task is a cybersecurity task.

20. A computer program product comprising a non-temporary computer-readable storage medium, wherein computer-readable instructions are stored in the computer-readable storage medium, and the computer-readable instructions are configured to perform the method according to any one of claims 1 to 19 when executed by at least one processor (204). Computer program products.

Images