Prompt engineering engine

Abstract

A system and method includes receiving a prompt specifying at least one task type; determining a system cue based on the received cue, the system cue including artificial intelligence (AI) system configuration details corresponding to the at least one task type; pre-processing the system hint to generate a pre-processed hint that includes code referenced in the system hint; sending the preprocessed prompt as an input prompt to an AI system; in response to a prompt that the AI system executes the preprocessing, receiving a result from the AI system; and storing a record of results from the AI system in a data repository.

Description

Technical Field

[0001] This application relates to a prompt engineering engine. Background Art

[0002] Application developers can typically design, create, deploy, and update programs for specific applications, operating systems, or platforms. In some cases, application developers may want to evaluate existing applications, seeking to, for example, add new features to them, complete updates, or reprogram existing applications. Developers may also write new code to meet the specifications of the application they are developing or previous applications. Different applications, operating systems, and platforms (i.e., computing environments) may require different code specific to that computing environment.

[0003] Some generative artificial intelligence (AI) systems, such as large language models (LLMs), are configured to generate code in response to prompts provided to them. LLMs typically accept prompts in natural language. Therefore, it is critical that the language used in the prompts to the LLM is configured so that the LLM can provide a response that includes correct and expected content.

[0004] In some cases, an application developer or other entity may want to investigate or understand various aspects of existing applications and programming codes, or create new programming code related to a computing environment. For example, a developer may want to understand some programming code previously written by others or generate new executable programming code to achieve a desired goal. Solutions to these technical problems can be improved by utilizing a large language model (LLM). Therefore, it is desirable to efficiently generate prompts related to the computing environment of the LLM, which elicit from the LLM an accurate desired response for the relevant computing environment. Summary of the invention

[0005] According to one aspect of the present invention, a system is provided, comprising: an inbound prompt engine application programming interface (API) for receiving a prompt, the prompt specifying at least one task type; a prompt selector for determining a system prompt based on the received prompt, the system prompt comprising artificial intelligence (AI) system configuration details corresponding to the at least one task type and comprising a combination of a description and at least one tag; a preprocessor for preprocessing the system prompt to generate a preprocessed prompt, the preprocessed prompt comprising code referenced in the system prompt; an outbound large language model API for sending the preprocessed prompt as an input prompt to an AI system and, in response to the AI ​​system executing the preprocessed prompt, receiving a result from the AI ​​system; and a data repository for storing a record of the results from the AI ​​system.

[0006] According to one aspect of the present invention, a computer-implemented method is provided, the method comprising: receiving a prompt, the prompt specifying at least one task type; determining a system prompt based on the received prompt, the system prompt comprising artificial intelligence (AI) system configuration details corresponding to the at least one task type and comprising a combination of a description and at least one tag; preprocessing the system prompt to generate a preprocessed prompt, the preprocessed prompt comprising a code referenced in the system prompt; sending the preprocessed prompt as an input prompt to an AI system; receiving a result from the AI ​​system in response to the AI ​​system executing the preprocessed prompt; and storing a record of the result from the AI ​​system in a data repository.

[0007] According to one aspect of the present invention, a non-transitory computer-readable medium is provided, storing instructions that, when executed by at least one processor, cause a computer to perform a method, the method comprising: receiving a prompt, the prompt specifying at least one task type; determining a system prompt based on the received prompt, the system prompt comprising artificial intelligence (AI) system configuration details corresponding to the at least one task type and comprising a combination of a description and at least one tag; preprocessing the system prompt to generate a preprocessed prompt, the preprocessed prompt comprising a code referenced in the system prompt; sending the preprocessed prompt as an input prompt to an AI system; receiving a result from the AI ​​system in response to the AI ​​system executing the preprocessed prompt; and storing a record of the result from the AI ​​system in a data repository. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Features and advantages of example embodiments and the manner in which the same are achieved will become more apparent with reference to the following detailed description taken in conjunction with the accompanying drawings.

[0009] Figure 1 It is an illustrative depiction of some typical systems of existing systems;

[0010] Figure 2 is an illustrative depiction of a system including a prompt engineering engine according to an example embodiment;

[0011] Figure 3 is an illustrative flow chart of a process according to an example embodiment;

[0012] Figure 4 is an illustrative depiction of a prompt library for a prompt engineering engine according to an example embodiment;

[0013] Figure 5 is an illustrative depiction of some aspects for determining system cues by a cue engineering engine according to an example embodiment;

[0014] Figure 6 is an illustrative depiction of some aspects related to a "reference finder" preprocessor of a prompt engineering engine according to an example embodiment;

[0015] Figure 7 is an illustrative depiction of some aspects related to a "code extractor" preprocessor of a hint engineering engine according to an example embodiment;

[0016] Figure 8 is an illustrative depiction of example preprocessing prompts according to an example embodiment;

[0017] Fig. 9 is an illustrative depiction of an example LLM response to a prompt provided by a prompt engineering engine according to an example embodiment;

[0018] Fig.10 is an illustrative depiction of some aspects related to a post-processor of a prompt engineering engine according to an example embodiment;

[0019] Fig.11 is an illustrative block diagram of an architecture associated with a prompt engineering engine according to an example embodiment;

[0020] Fig.12 is an illustrative block diagram including some detailed aspects of an architecture associated with a hint engineering engine, according to an example embodiment; and

[0021] Fig.13 is an illustrative block diagram of an apparatus or platform according to an example embodiment.

[0022] Throughout the drawings and detailed description, unless otherwise described, the same drawing labels will be understood to refer to the same elements, features, and structures. The relative sizes and depictions of these elements may be exaggerated or adjusted for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0023] In the following description, specific details are set forth in order to provide a thorough understanding of various example embodiments. It should be understood that various modifications to the embodiments will be apparent to those skilled in the art, and one or more principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. In addition, in the following description, many details are set forth for the purpose of explanation. However, it should be understood by those of ordinary skill in the art that embodiments may be practiced without using these specific details. In other examples, well-known structures, methods, processes, components, and circuits are not shown or described to avoid unnecessary details that obscure the description. Therefore, the present disclosure is not intended to be limited to the embodiments shown, but to conform to the widest scope consistent with the principles and features disclosed herein.

[0024] Figure 1is an illustrative depiction of a typical system of some existing systems. System 100 includes a user interface (UI) 105. User interface 105 can be implemented by one or more of a user device (e.g., a personal computer, a smart phone, etc.) or an application (e.g., a web browser) that is configured to receive prompts from a user or other entity (e.g., an application, a program, etc.) and send the prompts as input to a generative AI (artificial intelligence) or LLM (large language model) system 110. Typically, a generative AI or LLM system learns the patterns and structures of the training data used to train it and generates new data with similar properties. In some instances, a generative AI or LLM system 110 may be able to generate text, images, audio, and other media types using a generative model, depending on the training data used to train a particular system. In some embodiments, a generative AI or LLM system 110 can be trained on programming language code, thereby enabling the AI ​​system to generate new computer code having properties similar to the code on which it was trained. As Figure 1 As shown, the generative AI or LLM system 110 receives prompts from the UI 105 , and the generative AI or LLM system processes the received prompts and generates replies that are returned to the UI 105 .

[0025] In some contexts, application or platform developers may want to leverage some processing aspects of a generative AI or LLM system. For example, a developer may want to use some aspects of a generative AI or LLM system to generate code for certain tasks that the developer is working on. In some instances, a developer may want to evaluate existing code (e.g., legacy code written by someone else) to, for example, understand the existing code, add new features to the existing code, verify that the code complies with current coding standards / guidelines or practices, complete updates, etc. Developers may also want to generate new code according to certain specifications of the application for which they are responsible for developing code.

[0026] As a reference Figure 1In the example of, a developer working in a particular computing environment may provide a prompt to the AI ​​system 110 via the UI 105, "Create a database table named z_my_accounts with attributes id, description, account number", and expect the AI ​​system to return code including the attributes specified in the prompt and expressed in the programming language used in the computing environment where the developer is working. In this example, the AI ​​system may be GPT-3.5 or GPT-4 (Generative Pre-Trained Transformer, developed by OpenAI). In response to executing the prompt, the AI ​​system 110 returns code to generate a database table with the attributes specified in the prompt. However, the returned code is not the programming language used in the computing environment where the developer is working. For example, the returned code may be SQL (Structured Query Language), while the programming language used in the developer's computing environment is ABAP (Advanced Business Application Programming, developed by SAP SE). The returned code does not include the correct content in the format expected by the developer. In some aspects, the AI ​​system 110 does not understand what the developer wants via the prompt, or the AI ​​system lacks certain contextual information.

[0027] Figure 2 200 includes an application or developer tool (e.g., an integrated development environment) including a user interface (UI) 205. UI 205 may be configured to receive prompts from a user. In some cases, the application or developer tool may be configured to automatically generate prompts at least in part based on the interaction of the user with the application or developer tool. The prompt is sent to prompt engineering engine 210 via UI 205 as input to the prompt engineering engine. As will be described in more detail below, prompt engineering engine 210 processes the prompt to determine the pre-processed prompt (e.g., the pre-processed prompt is determined based on the initial prompt) sent to generative AI or LLM system 215. As used herein, generative AI or LLM system 215 may be interchangeably referred to as LLM. Generative AI or LLM system 215 may be able to generate text, images, audio, other media types, and programming language code using a generative model based on training data for training the LLM system. Typically, the pre-processed prompts include configuration details and context information to enable the LLM system 215 to efficiently return replies that accurately include the details requested in the prompt. Replies based on the pre-processed prompts can be further processed by the prompt engineering engine to generate post-processed replies that are sent to the application or developer tool. The post-processed replies accurately include the content details of the format or configuration requested in the initial prompt.

[0028] As a reference Figure 2 In the example of, a developer working in a particular computing environment ABAP (or other programming language) can provide a prompt via UI 205 to prompt the engineering engine 210, "Create a database table named z_my_accounts with attributes id, description, account number." In this example, the developer expects a response including ABAP code with the attributes specified in the prompt and accurately represented (e.g., syntax, format, etc.) in the programming language used in the computing environment in which the developer is working. The prompt engine can pre-process the prompt to generate a pre-processed prompt, where the pre-processed prompt includes the details of the LLM 215 (e.g., instructions, rules, parameter values, context information, etc.) to provide a reply including code that satisfies the task requested in the initial prompt. The reply from the LLM 215 can be processed by a post-processor of the prompt engineering engine 210 to generate a post-processed reply including the expected response (i.e., the correct content configured according to the request) to fully comply with the request and task of the initial prompt.

[0029] As another example, a developer (or other entity) may want to understand some existing code that they have not yet written. Although the ability such as debugging or reading code may be a way for developers to understand code, the present disclosure provides a mechanism for automatically interpreting subject code using a large language model (i.e., typically, AI). In this example, the developer wants to interpret a certain ABAP CDS view (e.g., " / DMO / R_TRAVEL_D"), where the CDS view is a part of ABAP created using a data definition language (Data Definition Language, DDL) to build or define a data model. In this example, the CDS view " / DMO / R_TRAVEL_D" is an ABAP code artifact known to the ABAP system that the developer wants to interpret (i.e., what does this view do?).

[0030] In one example, a prompt (“Explain CDS View / DMO / R_Trave_D”) is submitted to an LLM (e.g., GPT-4) such as Figure 1as shown. The LLM has no connection to the developer's ABAP system and is unaware of the view " / DMO / R_TRAVEL_D". In response to the prompt, the LLM may reply with a response that merely states that the view " / DMO / R_TRAVEL_D" is related to travel, indicating that the exact code of the subject ABAP CDS view is unknown because it is system defined in the CDS and SAP systems. In such an instance, the LLM may have some limited idea of ​​what the view " / DMO / R_TRAVEL_D" is, but it cannot provide the exact details of the CDS view to "explain" the view requested in the prompt. In some instances where the LLM does not know the code, the LLM may present incorrect information (i.e., AI hallucination) in response to the prompt.

[0031] In another example, a prompt ("Explain CDS View / DMO / R_TRAVEL_D") may be submitted to a Figure 2 , which adds certain information to the initial hint to generate a pre-processed hint that is presented and processed by the LLM. In turn, the LLM provides a reply that may include a detailed line-by-line description or explanation of what the view (i.e., code artifact) " / DMO / R_TRAVEL_D" actually does. In some instances, the reply from the LLM may be further post-processed to configure the reply as specified in the initial hint.

[0032] In some aspects, the present disclosure provides mechanisms suitable for a variety of use cases, including functionality to achieve different desired tasks or results, such as, but not limited to, for example, interpreting code, tables, and other programming language artifacts and features; generating code, tables, and other programming language artifacts and features; etc.

[0033] In some aspects, the disclosed prompt engineering engine can perform certain functions including, but not limited to, interpreting code, generating tables, interpreting views, interpreting behaviors, etc. In some embodiments, instructions can be determined for the LLM in the form of prompts or system prompts (e.g., configuration details and information used by the LLM) to elicit good quality results from the LLM in response to user prompts.

[0034] Figure 3is an illustrative flow chart of a process 300 according to an example embodiment. In some embodiments, the prompt engineering engine herein may be applied, integrated, or otherwise used in an ABAP computing environment, and in such embodiments is referred to as an ABAP prompt engineering engine. However, in some embodiments, the prompt engineering engine herein may be applied, integrated, or otherwise used in another computing environment other than ABAP. In some aspects, process 300 provides an overview of the process of executing and applying a prompt engineering engine as disclosed herein.

[0035] At operation 305, the user (or other entity, such as, for example, a system, application, or developer tool) specifies a prompt. Continuing the example introduced above, the prompt received at operation 305 is "Explanation View / DMO / R_TRAVEL_D". In this example, the user (or other entity) wants to explain the code artifact (that is, the view called " / DMO / R_TRAVEL_D") in the user's system. In some aspects, the prompt can specify at least one task type or the type of task to be completed. In this example, the prompt "Explanation View / DMO / R_TRAVEL_D" specifies the task type of the explanation view. Other task types can include, for example, creating a table, a view, or other code entities or features.

[0036] At operation 310, the initial prompt may be augmented with additional information to determine a "system prompt." The system prompt is determined based on the initial prompt and should include information such that when the system prompt is provided as input to the LLM, the LLM or other generative AI (e.g., GPT-4) has an improved chance of actually returning an accurate and complete (i.e., good) result relative to the initial prompt.

[0037] As part of determining a "system prompt" as indicated at operation 310, a prompt library 315 may be referenced, called, or otherwise used to determine a system prompt. In some embodiments, for each task type supported by the prompt engineering engine herein (e.g., interpreting code, views, tasks, etc.; creating tables, rules, etc.; creating CDS views, rules, etc.; generating data generators, rules, etc.; etc.), there is a corresponding "system prompt" (e.g., system settings or configuration details). In some aspects, a system prompt may refer to configuration details of the LLM, which may be used to initialize communications with the LLM, which may include certain rules and instructions for the LLM to use when executing the prompt so that the LLM can better understand or know what type of response is being requested of it. In this way, there may be different "system prompts" for different task types (i.e., as specified in the user-provided "prompt" at operation 305).

[0038] As an example, there may be a system prompt for each of the task types "Explain Code", "Create Table", "Create CDS View", "Create Application", "Generate Data", etc. In some aspects, the system prompts are extensible so that for each use case (i.e., task type), a corresponding "system prompt" can be determined or generated. In some embodiments, the system prompts stored in the prompt library 315 or generated based on the features therein can be edited (e.g., modified, added, deleted, etc.) and used to determine the system prompt associated with the current or future prompt.

[0039] In some embodiments, the LLM used to determine the system prompt at operation 310 may be selected or otherwise determined based on the LLM's ability or proficiency to, for example, understand the intent of the initial user (or other entity) prompt received at operation 305 and use that initial prompt to determine a system prompt that may lead to an accurate result when the system prompt is further provided as input to the LLM (e.g., Figure 3 , operation 335). In some embodiments, the LLM used to determine the suitability of the system prompt herein may be based at least in part on the model and the training data used to train the LLM. In some aspects, the LLM used to determine the system prompt at operation 310 may be able to understand or distinguish from the initial prompt what the initial prompt is about or what is intended to be done (i.e., the task type), and further compare with the prompt library (e.g., Figure 3 , 315) collaborate to determine appropriate system prompts. In some embodiments, the LLM can be specially configured, trained or otherwise adapted to perform the determination of system prompts herein.

[0040] In some aspects, after determining the system prompt at operation 310, the remainder of process 300 may be completely unaware of the LLM or AI used to perform the disclosed operations (e.g., operations 325, 335, and 345). For example, in some embodiments, the LLM used to determine the system prompt at operation 310 may be different from the LLM executed at operation 335 to generate the expected response at 340. In some embodiments, the LLM or AI used when performing operations after determining the system prompt may be selected or otherwise determined based on, for example, a specific use case or implementation of an application. As an example, a specific use case may involve language processing for which a specific LLM may be particularly well suited. Although this specific LLM may be particularly well suited for language-related use cases and is selected for use in this example at operation 335, the framework depicted by process 300 (or elsewhere herein) is not limited to any specific LLM.

[0041] In some embodiments, the prompt library 315 can support a preprocessor and a postprocessor, wherein the preprocessor is used to process the prompt before the prompt is presented to the LLM, and the postprocessor is used to process the reply from the LLM. In some embodiments, the preprocessor and the postprocessor are provided as part of the prompt engineering engine (e.g., as part of the prompt library in ABAP). In some aspects, the preprocessor and the postprocessor can be defined by a user of the prompt engineering engine.

[0042] In some embodiments, a user (eg, a developer or other authorized entity) can define configuration details and rules specified by system prompts and define pre-processors and post-processors.

[0043] Figure 4 4 is an illustrative depiction of a prompt library 400 for prompting an engineering engine according to an example embodiment. As shown, prompt library 400 includes system prompts 405, pre-processors 410, and post-processors 415. Figure 4 In the example of , the listed system prompts include system prompts 420 for task types such as interpreting codes, rules, etc.; system prompts 425 for task types such as creating tables, rules, etc.; system prompts 430 for task types such as creating CDS views, rules, etc.; system prompts 435 for task types such as creating RAP (RESTful application programming model) applications, rules, etc.; system prompts 440 for other task types; and system prompts 445 for task types such as generating data. Some embodiments may include more than Figure 4 More, fewer, or other system prompts explicitly shown in the system prompts.

[0044] In hint library 400, the pre-processors listed include reference finder 450 and code extractor 455. The post-processors shown include RAP generator 460. Figure 4 The pre-processors and post-processors depicted in are examples of some embodiments and are not limitations on the scope of pre-processors and post-processors suitable for use with the hint engineering engine disclosed herein.

[0045] Referring again to the example prompt of "Explain View / DMO / R_TRAVEL_D", operation 310 of determining relevant system prompts may be accomplished by requiring the AI ​​to select one prompt from a given set of prompts (e.g., a list of prompts in a "prompt library" 315) that best matches what the prompt requests (i.e., the task to be completed). In this example, operation 310 may include, for example, populating a system prompt template from a list of system prompt templates from a prompt library with information specified in the initial prompt. At operation 310, the AI ​​may determine based on the initial prompt that the user wants to "Explain Code", and select a specific "rule" from the prompt library 315 that is related to the specific task of explaining code, as well as a pre-processor and post-processor corresponding to the determined task type. The system prompts and pre-processors as determined at operation 310 are depicted at 320.

[0046] Figure 5 is an illustrative depiction of determining system prompts by a prompt engineering engine according to an example embodiment. Figure 5 As shown, operation 510 of determining a system prompt based on prompt 505 (e.g., "Interpretation View / DMO / R_TRAVEL_D") can cause the prompt engineering engine of this article to select system prompt 520, possibly with the help of LLM, and system prompt 520 includes prompt template 515 that can be filled with information from prompt 505.

[0047] In this example, process 300 has determined that the prompt engineering engine is being tasked with "explaining" something. Exactly what needs to be explained will be determined further. At operation 325, a preprocessor associated with the system prompt at 320 is executed to find, for example, a reference to an existing code artifact in the ABAP system (i.e., computing environment). In some embodiments, operation 325 of finding a reference in the ABAP system can be supported by a reference finder using LLM.

[0048] In some embodiments, an LLM (e.g., GPT-4 or other AI) can help determine system cues and their characteristics, including pre-processors and post-processors associated with a given system cue.

[0049] Figure 6 is an illustrative depiction of some aspects related to a "reference finder" preprocessor that prompts an engineering engine, according to an example embodiment. Figure 6, a pre-processor “reference finder” 605 of the hint library is used to find or otherwise determine the code reference in the hint 615 (e.g., “explain view / DMO / R_TRAVEL_D”) and return the result in a structured format (e.g., JSON notation), as shown at 610. As shown at 620, the reference finder 605 can return an enhanced hint of a “view” type that “explains” the name “ / DMO / R_TRAVEL_D” (i.e., the referenced code “ / DMO / R_TRAVEL_D”), which is specified in the specified format (i.e., a list) in the original user-provided hint.

[0050] Continuing with this example, a preprocessor called a "code extractor" can be used to extract the code of the view (" / DMO / R_TRAVEL_D") from the ABAP system (or other related computing environment). The operation of determining the system prompt in the current example has been determined by the reference finder as described above that the item " / DMO / R_TRAVEL_D" in the initial prompt is a code artifact that should be in the computing environment of the prompt engineering engine (i.e., the ABAP computing environment in this example).

[0051] Figure 7 is an illustrative depiction of some aspects related to a "code extractor" preprocessor of a hint engineering engine according to an example embodiment. Figure 7 In the example, the “code extractor” 705 of the prompt library is used to fetch or otherwise retrieve the code of the view “ / DMO / R_TRAVEL_D” from the ABAP system, as shown at 710. At 715, the code fetched from the ABAP system code by executing the code extractor may be presented.

[0052] refer to Figure 3 , Figure 6 and Figure 7 The operations depicted in 324 may be associated with operation 325 (execution of a preprocessor associated with the determined system prompt) to generate or otherwise determine a preprocessed prompt 330. In this example, the preprocessors executed include a reference finder and a code extractor. Other preprocessors may be selected and executed based on the determined system prompt and an initial prompt associated with the determined system prompt.

[0053] Figure 8 is an illustrative depiction of an example pre-processed prompt 800 according to an example embodiment. Figure 8 The example pre-processed hint 800 shown in FIG. 8 includes an initial hint 805 , a determined system hint 810 including configuration details of the LLM, and related code 815 referenced and retrieved by the preprocessor associated with the determined system hint.

[0054] After determining, generating, or otherwise assembling the preprocessed prompt 330, process 300 continues to execute the preprocessed prompt with the help of the LLM at operation 335. The preprocessed prompt can be submitted as input to the LLM. In this example, the user intent from the initial prompt (e.g., "Explanation View / DMO / R_TRAVEL_D"), additional rules from the prompt library for the specified task (i.e., system prompts), and code referenced in the prompt and retrieved from the ABAP system are included in the preprocessed prompt. The LLM is called at operation 335 to execute using the preprocessed prompt 330 to generate a related response or result 340. The result 340 can be stored in a data repository or other data storage device or system.

[0055] Fig. 9 is an illustrative depiction of an example LLM response 900 to a prompt provided by a prompt engineering engine, according to an example embodiment. Fig. 9 As shown, the response from the LLM includes a complete description or explanation of what the " / DMO / R_TRAVEL_D" view specified in the initial prompt is, including, for example, a line-by-line explanation of what the code associated with the specified view does or how it functions. Note that example LLM response 900 illustratively depicts a portion of a possible response from the LLM.

[0056] Reference again Figure 3 At operation 345, one or more post-processors may be executed to further process the results returned by the LLM to generate a final result received at operation 350. Similar to the pre-processors, the post-processors executed at operation 345 may be specific to or related to a particular task specified in the initial prompt received from the user (or other entity).

[0057] Fig.10 is an illustrative depiction of some aspects related to a post-processor of a prompt engineering engine according to an example embodiment. Fig.10 In the example of , an initial prompt 1005 (e.g., "Create sample data for a database table ...") includes a request or task to write a natural language description to create sample data for a database table. Prompt 1005 can cause the prompt engineering engine of this document to determine or otherwise generate a pre-processed prompt 1010, which includes system prompts and additional information, where the LLM is required to return a JSON (JavaScript Object Notation) document containing the generated sample data, such as Fig.10As shown. The prompt to perform preprocessing by the LLM can produce a JSON document 1015. Based on the task(s) specified in the initial prompt 1005 and the post-processor(s) associated with the determined system prompt, a post-processor (e.g., a "data generator") can be executed to transform the JSON document 1015 of the LLM result into the format or configuration specified in the system prompt. In this example, a post-processor of type data generator can be called to process the JSON description of the application to create ABAP code containing methods to insert the generated data into the database table, as shown in 1020.

[0058] In some embodiments, the LLM or AI used when performing operations (e.g., operations 325, 335, and 345) after determining the system prompt can be selected or otherwise determined based on, for example, a specific use case or implementation of the application. In one or more instances, the LLM can be selected from the following list of LLMs, which includes GPT models (e.g., 3.5Turbo and 4) provided by OpenAI; Luminous from Aleph Alpha; BLOOM (BigScience Large Open Science Multilingual Language Model) via Huggingface; Claude from Anthropic; Falcon from Technology Innovation Institute; Llama model from Meta AI; Command from Cohere; and PaLM developed by Google AI. The foregoing list is not intended to be exhaustive, but rather to illustrate some types and kinds of LLMs that can be used in accordance with the present disclosure.

[0059] Fig.11 1100 is an illustrative block diagram of an architecture associated with a prompt engineering engine according to an example embodiment. In some embodiments, architecture 1100 is a high-level architecture in which some components including applications or tools 1110, prompts 1150, and processors 1155 may be application-specific parts that can be implemented in ABAP, which will use an ABAP system (e.g., ABAP system 1105). These components may be specified content of an ABAP system application (e.g., SAP ERP, enterprise resource planning software / system developed by SAP SE, SAP S / 4HANA developed by SAP SE, etc.). In some embodiments, an ABAP (or other computing environment application depending on the context and use case) application may specify, generate, or determine system prompts 1150 and pre- and post-processors 1155, and include application code for performing such functions.

[0060] As an example, in an instance where the application 1110 wants to call certain functions including interaction with the LLM 1115, the application 1110 may call the inbound prompt engine API 1125 of the ABAP prompt engineering engine 1120. The inbound prompt engine API 1125 may be configured to support communication between the computing environment application or tool 1110 and the ABAP prompt engineering engine 1120. In response to receiving the prompt, a call may be issued to the prompt engine orchestrator 1130 that coordinates the call to the prompt selector 1135. The prompt selector 1135 may be configured to determine what task to perform based on the task type derived from the prompt received from the application or tool 1110. The prompt selector 1135 may determine the task to be performed (i.e., determine the system prompt) via interaction with the prompt library 1145. The determined (one or more) system prompts may be provided to the prompt engine orchestrator 1130, wherein the prompt engine orchestrator may execute the provided (one or more) system prompts (one or more system prompts, including processes defined as including multiple system prompts in some instances) via the prompt runner 1140. In some aspects, the hint runner 1140 can be operated to call the LLM 1115 via the outbound LLM API 1170. The outbound LLM API 1170 can be configured to facilitate communication between the ABAP hint engineering engine 1120 and the LLM 1115. In some embodiments, the ABAP hint engineering engine 1120 collects traces using a trace collector 1160 when the ABAP hint engineering engine executes. In some aspects, the trace collector 1160 collects or saves (e.g., in a data storage device 1165) a record or indication of content sent to the LLM 1115 (e.g., pre-processed hints, etc.) and content received from it (e.g., LLM reply results to processing pre-processed hints). The hint engineering engine can also process the results received from the LLM 1115 using one or more post-processors 1155, wherein the hint engine orchestrator 1130 can support post-processing execution of the processor 1155.

[0061] Fig.12 is an illustrative block diagram including some detailed aspects of an architecture 1200 associated with a prompt engineering engine according to an example embodiment. Specifically, the architecture 1200 includes Fig.12 In some embodiments that include computing systems other than the ABAP platform, the computing environment may be different from Fig.12 In some embodiments, architecture 1200 includes a prompt library ( Fig.121205, the system prompts 1210, and the pre-processors and post-processors 1215. In some aspects, the ABAP system 1205, the system prompts 1210, and the pre-processors and post-processors 1215 may be similar to Fig.11 The ABAP system 1105, system prompts 1150, and pre- and post-processors 1155 depicted in FIG. 1 include similar functionality. Fig.11 on the contrary, Fig.12 Additional descriptive details are provided regarding the illustrated system prompts 1210 and pre- and post-processors 1215 .

[0062] In some embodiments, prompt 1210 includes a name, a description, and one or more labels. Based on these attributes, prompts can be retrieved from a prompt library. Therefore, if a certain type of task is to be performed, in some embodiments, with the help of AI, the prompt engineering engine of this article can determine or select a system prompt corresponding to the task type by, for example, a combination of the name, description, and label of the system prompt. As also shown in the figure, the system prompt 1210 may include certain other contents about the prompt, such as, for example, a rule set and additional text information. The system prompt may include one or more parameters. In some instances, the system prompt may point to the next system prompt to be executed after the current subject system prompt. The "next prompt" attribute or feature of the system prompt can be used to point to another system prompt. For example, a complex task may be decomposed into simpler tasks and processed continuously. In one case, an outbound LLM API can be called with a first system prompt, the result of the first system prompt can be received and retained, and then a call to the next system prompt is issued to the outbound LLM API, and so on. This can be an example of linking prompt execution. The next prompt parameter can be used to, for example, implement a system prompt execution loop, in which the system prompt points to itself or other system prompts. To avoid running an infinite loop, the execution loop can be prompted by a postprocessor exit, which checks the results after each iteration and stops execution if a certain condition is met.

[0063] In some embodiments, the system prompts herein may have an error retry specification for use in the event that an error is encountered while processing the system prompt, wherein the error retry specification may specify a number of executions of the retry system prompt and also include rules for providing error information to the LLM. The LLM may use the error information to attempt to repair or correct the error.

[0064] In some embodiments, system prompts 1210 may be defined or otherwise indicated as searchable. In some embodiments, system prompts 1210 may be stored in a database table (or other data structure), where the system prompts may be specified or otherwise defined as searchable or non-searchable. As used herein, a searchable system prompt may be searchable by a combination of its name, description, and tag in an associated database table. For example, a system prompt selection process used by the prompt engineering engine of the present invention (see, e.g., discussion of process 300 and architecture 1100) may automatically select an appropriate system prompt from a prompt library, where a system prompt defined as searchable may be found in the process of selecting a prompt from the prompt library. In some embodiments, a searchable prompt of the present invention may be called from an outside (i.e., external) source. In instances where a system prompt is not defined as searchable, then this type of system prompt may only be used by calling it directly (i.e., specified by name). In some embodiments, a system prompt may be accessed through an inbound prompt engine API (e.g., Fig.11 , inbound prompt engine API 1125) directly calls the non-searchable prompt. The option of non-searchable system prompt provides a mechanism for directly addressing non-searchable system prompts. In the case of directly calling non-searchable system prompts, the system prompt determination process involving the prompt library in this article can be bypassed.

[0065] The system prompt may also include a list of corresponding pre-processors and post-processors 1215 associated therewith. In some embodiments, the pre-processors and post-processors 1215 may be registered or otherwise associated with the system prompt. In this way, when a particular system prompt is executed, the pre-processors and post-processors defined as being associated therewith are also executed. In some aspects, the pre-processors and post-processors may be method calls to some other code that may be implemented by the ABAP application. For example, Fig.12 As shown, (one or more) pre-processors and post-processors can be defined as having names and certain functions that can be represented in ABAP code (or other computing environment appropriate code). ABAP code can be called from an ABAP system application. In some instances, the ABAP code can access application data on the ABAP system from an internal data source 1220. In some cases, the ABAP code can be executed to call out an external data source 1225 (e.g., a search engine, a document store, etc.) to obtain additional information that can be used to complete the task to be performed.

[0066] In some embodiments, process 300 (e.g., a computer-implemented method or process) and architectures 1100 and 1200 provide a framework for implementing various applications, including, for example, applications and use cases involving different hints, multiple hints, iterative hints, one or more different pre-processors and post-processors; use cases including one hint calling the execution of another hint, splitting complex hint tasks into smaller or multiple tasks); etc. Some other use cases in which the hint engineering engine may be applied herein may include, for example, code generation to generate code documentation (e.g., coding style guides, etc.); an automatic code review tool that, based on some clean code guidelines, can provide recommendations on how to improve the reviewed code (e.g., analyze code changes and provide recommendations on, for example, "better" naming options for variables and other aspects that are subject to code style review); selecting and executing a post-processor to check the code returned in the LLM reply to determine if there are violations of some rules of the "clean code" guidelines; and other use cases.

[0067] Fig.13 is an illustrative block diagram of an apparatus or platform according to an example embodiment. Note that the embodiments described herein may be implemented using any number of different hardware configurations. For example, Fig.13 is possible, for example, with Fig.11 and Fig.12 1100 and 1200 (and any other systems described herein) are associated with the apparatus or platform 1300. The platform 1300 includes a processor 1305, such as one or more commercially available CPUs in the form of a single-chip microprocessor, and a main memory 1325, the processor 1305 being coupled to a computer configured to communicate via a communication network ( Fig.13 The platform 1300 also includes a communication device 1310 (not shown) for communicating with the platform 1300. The communication device 1310 can be used to communicate with one or more remote user devices, networks or entities 1365, for example, via a communication interface 1360 and a communication path 1370. The platform 1300 also includes an input device 1315 (e.g., a computer mouse, keyboard, etc.) and an output device 1320 (e.g., a computer monitor for drawing a display, sending recommendations or alerts, creating monitoring reports, etc.). According to some embodiments, mobile devices, PCs, and other devices can be used to exchange data with the platform 1300.

[0068] Processor 1305 also communicates with storage device 1330. Storage device 1330 may be implemented as a single database, or different components of storage device 1330 may be distributed using multiple databases (i.e., different deployment data storage options are possible). Storage device 1330 may include any suitable data storage device, including a combination of magnetic storage devices (e.g., hard drive 1335), optical storage devices, mobile phones, and semiconductor memory devices. Storage device 1330 may store programs, which may be shared with removable storage unit 1345 for controlling processor 1305. Processor 1305 executes instructions of one or more programs stored on one or more of storage devices 1340, 1345, and 1350, which may be executed to perform operations in accordance with any of the embodiments described herein (e.g., Figure 3-Figure 10 ) to perform the operation.

[0069] The programs may be stored in compressed, uncompiled, encrypted, and other configuration formats. The stored programs may also include processor executable code and other program elements, such as an operating system, a clipboard application, a database management system, and device drivers used by the processor 1305 to interface with peripheral devices. In some embodiments (such as Fig.13 In the embodiment shown in ), the storage device 1330 also stores data on the removable storage unit 1350 via the interface 1355.

[0070] As used herein, data may be "received" or "sent" by, for example: (i) platform 1300 from another device; or (ii) a software application or module within platform 1300 from another software application, module, or any other source.

[0071] As will be understood based on the foregoing description, the above examples of the present disclosure may be implemented using computer programming or engineering techniques including computer software, firmware, hardware, or any combination or subset thereof. Any such obtained program having computer readable code may be embodied or provided in one or more non-transitory computer readable media, thereby manufacturing a computer program product, i.e., an article of manufacture, according to the discussed examples of the present disclosure. For example, a non-transitory computer readable medium may be, but is not limited to, a fixed drive, a disk, an optical disk, a tape, a flash memory, an external drive, a semiconductor memory (such as a read-only memory (ROM), a random access memory (RAM) (such as main memory 1325), and any other non-transitory sending or receiving medium (such as the Internet, cloud storage, the Internet of Things (IoT) or other communication networks or links). Articles of manufacture containing computer code may be manufactured and used by directly executing the code from one medium, by copying the code from one medium to another, or by transmitting the code over a network.

[0072] A computer program (also referred to as a program, software, software application, "app" or code) may include, for example, computer-readable machine instructions for a programmable processor, and may be implemented in high-level procedural, object-oriented programming languages, assembly / machine languages, etc. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, cloud storage, Internet of Things, and device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. However, "machine-readable medium" and "computer-readable medium" do not include transient signals. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and any other kind of data to a programmable processor.

[0073] The above description and illustration of the process herein should not be considered to imply a fixed order for performing the process steps. Instead, the processing steps can be performed in any feasible order, including performing at least some steps simultaneously. Although the present disclosure has been described in conjunction with specific examples, it should be understood that various changes, substitutions and modifications that are obvious to those skilled in the art may be made to the disclosed embodiments without departing from the spirit and scope of the present disclosure set forth in the appended claims.

Claims

1. A system comprising: An inbound prompt engine application programming interface API for receiving a prompt, the prompt specifying at least one task type; a prompt selector for determining a system prompt based on the received prompt, the system prompt comprising artificial intelligence (AI) system configuration details corresponding to the at least one task type and comprising a combination of a description and at least one tag; a preprocessor for preprocessing the system prompt to generate a preprocessed prompt, the preprocessed prompt including a code referenced in the system prompt; an outbound large language model API for sending the pre-processed prompt as an input prompt to an AI system and, in response to the AI ​​system performing the pre-processed prompt, receiving a result from the AI ​​system; as well as A data repository for storing records of results from the AI ​​system.

2. The system according to claim 1, wherein: The prompt is received from at least one of a user, an executable application, and an application development environment.

3. The system according to claim 1, wherein: Determination of the system prompt includes: the prompt selector references a prompt library, the prompt library including at least one of the following items: a set of rules corresponding to the at least one task type, a set of system prompt templates in which each system prompt template corresponds to each task type in the at least one task type, and a set of system prompt templates in which one or more system prompt templates correspond to each task type in the at least one task type.

4. The system of claim 1, further comprising at least one pre-processor determined by the prompt engine orchestrator to perform pre-processing of the system prompts, the at least one pre-processor being determined based on the at least one task type.

5. The system according to claim 1, further comprising: post-processing, by a prompt engine scheduler, the result received from the AI ​​system to generate a post-processed result, wherein the post-processed result is configured to be specified by a post-processor determined based on the at least one task type; as well as A record of the results of the post-processing is stored in the data repository.

6. The system according to claim 1, wherein: The code referenced in the system prompts includes the Advanced Business Application Programming (ABAP) programming language.

7. The system according to claim 1, wherein: The task type includes at least one of a task for interpreting a designated code artifact and a task for creating a designated code artifact.

8. A computer-implemented method, the method comprising: receiving a prompt, the prompt specifying at least one task type; determining a system prompt based on the received prompt, the system prompt comprising artificial intelligence (AI) system configuration details corresponding to the at least one task type and comprising a combination of a description and at least one tag; preprocessing the system prompt to generate a preprocessed prompt, the preprocessed prompt including a code referenced in the system prompt; sending the preprocessed prompt as an input prompt to an AI system; In response to the prompting of the AI ​​system to perform the preprocessing, receiving a result from the AI ​​system; as well as A record of the results from the AI ​​system is stored in a data repository.

9. The method according to claim 8, wherein: The prompt is received from at least one of a user, an executable application, and an application development environment.

10. The method according to claim 8, wherein: Determining the system prompt includes referencing a prompt library, the prompt library including at least one of the following: a set of rules corresponding to the at least one task type, a set of system prompt templates in which each system prompt template corresponds to each task type in the at least one task type, and a set of system prompt templates in which one or more system prompt templates correspond to each task type in the at least one task type.

11. The method according to claim 8, further comprising: At least one pre-processor for completing pre-processing of the system prompt is determined, the at least one pre-processor being determined based on the at least one task type.

12. The method according to claim 8, further comprising: post-processing the result received from the AI ​​system to generate a post-processed result, the post-processed result being configured to be specified by a post-processor determined based on the at least one task type; as well as A record of the results of the post-processing is stored in the data repository.

13. The method according to claim 8, wherein: The code referenced in the system prompt includes the Advanced Business Application Programming (ABAP) programming language.

14. The method according to claim 8, wherein: The task type includes at least one of a task for interpreting a designated code artifact and a task for creating a designated code artifact.

15. A non-transitory computer readable medium storing instructions which, when executed by at least one processor, cause a computer to perform a method comprising: receiving a prompt, the prompt specifying at least one task type; determining a system prompt based on the received prompt, the system prompt comprising artificial intelligence (AI) system configuration details corresponding to the at least one task type and comprising a combination of a description and at least one tag; preprocessing the system prompt to generate a preprocessed prompt, the preprocessed prompt including a code referenced in the system prompt; sending the preprocessed prompt as an input prompt to an AI system; In response to the prompting of the AI ​​system to perform the preprocessing, receiving a result from the AI ​​system; as well as A record of the results from the AI ​​system is stored in a data repository.

16. The medium according to claim 15, wherein The prompt is received from at least one of a user, an executable application, and an application development environment.

17. The medium according to claim 15, wherein Determining the system prompt includes referencing a prompt library, the prompt library including at least one of the following: a set of rules corresponding to the at least one task type, a set of system prompt templates in which each system prompt template corresponds to each task type in the at least one task type, and a set of system prompt templates in which one or more system prompt templates correspond to each task type in the at least one task type.

18. The medium of claim 15, further comprising determining at least one pre-processor for completing pre-processing of the system prompt, the at least one pre-processor being determined based on the at least one task type.

19. The medium of claim 15, further comprising: post-processing the result received from the AI ​​system to generate a post-processed result, the post-processed result being configured to be specified by a post-processor determined based on the at least one task type; as well as A record of the results of the post-processing is stored in the data repository.

20. The medium of claim 15, wherein the task type comprises at least one of a task for interpreting a designated code artifact and a task for creating a designated code artifact.

Images