Object-oriented and / or symbolic prompting to impose structure on generative model output

Abstract

Implementations are described herein for object-oriented and / or symbolic prompting to impose structure on generative model output. In various implementations, an input prompt for a generative model may be assembled to include an input object and a desired output type. The input object may express a task to be performed using the generative model. The desired output type may include one or more object-oriented constructs, composed using an object-oriented programming language, which define structure to impose on generative model output predicted using the generative model. The input prompt may be processed using the generative model to predict an output object. The output object may be structured in accordance with one or more of the object-oriented constructs.

Description

OBJECT-ORIENTED A D / OR SYMBOLIC PROMPTING TO IMPOSE STRUCTURE ON GENERATIVE MODEL OUTPUTBackground

[0001] Content generated using generative artificial intelligence (Al) models such as large language models (LLMs) is typically unstructured. For instance, LLMs are often used to generate natural language output based on natural language input. Moreover, generative models such as LLMs are often considered non-determini stic and / or stochastic because the output they generate is non-deterministic (e.g., the same input prompt may yield multiple different outputs), e.g., due to random seeding. For example, when LLMs are prompted with the question “compute 1 + 1,” the LLM output may include responses such as “2”, “two”, “the answer is 2,” and so on.Summary

[0002] It is possible to use prompt engineering to impose at least some structure and / or consistency on generative model output. For example, heuristics and / or rules-based systems may be integrated with generative models such as LLMs to facilitate this process. However, the intricate interplay between heuristics / rules-based systems and generative model(s) may involve nuanced prompt engineering and / or object parsing, which, due to factors such as the fluid nature of natural language, may render the interactions fragile. In a workflow (e.g., one or more computer programs) that incorporates a generative model, operation(s) downstream from the generative model may be configured to receive input that has very specific structure, e.g, dictated by one or more schemas. If output predicted by the generative model does not reliably comport with these schemas, the entire workflow may be compromised.

[0003] Implementations described herein relate to formulating generative model invocations as object-to-object mappings (extending beyond traditional text-to-text mappings) using object- oriented constructs as schema in generative model input prompts. More particularly, but not exclusively, techniques described herein enable the use of familiar object-oriented programming constructs such as classes, types, and / or objects to prompt (or “condition”) generative models to generate output that is structured and / or consistent. For instance, a user can cause an LLM to predict a solution to a math problem along with a step-by-step resolution by defining a class with steps and answer fields.

[0004] In various implementations, a method may be implemented using one or more processors and may include: assembling an input prompt for a generative model, the input prompt comprising an input object and a desired output type, wherein the input object expresses a task to be performed using the generative model, and wherein the desired output type comprises one or more object-oriented constructs, composed using an object-oriented programming language, that define structure to impose on generative model output predicted using the generative model; and processing the input prompt using the generative model to predict an output object, wherein the output object is structured in accordance with one or more of the object-oriented constructs.

[0005] In various implementations, the input object may express the task to be performed using natural language and / or using the object-oriented programming language. In various implementations, the task to be performed may be an analytic task and / or a mathematical task.

[0006] In various implementations, the method may further include passing the output object as input to a downstream process, and the downstream process may be configured to receive input that is structured in accordance with one or more of the object-oriented constructs. In various implementations, the input prompt may further include at least one exemplar. The at least one exemplar may include an exemplar input object, an exemplar output type, and an exemplar output object that is structured in accordance with the exemplar desired output type. In various implementations, the exemplar output type may include one or more additional object-oriented constructs composed using the object-oriented programming language.

[0007] In various implementations, the method may further include processing data indicative of a natural language snippet that expresses the task using the same generative model or a different generative model to predict the input object and the desired output type. In various implementations, the input object may include one or more statements, composed in the object- oriented programming language, that set a value of a string variable to a natural language snippet that expresses the task. In various implementations, the method may include training the same generative model or a different model based on the natural language snippet and the desired output type.

[0008] In various implementations, the input object may include a plurality of input parameters for performance of the task. In various implementations, the plurality of input parameters may include a first input parameter in a first modality and a second input parameter in a second modality that is different than the first modality. In various implementations, the first modalitymay be structured text or natural language, and the second modality may be one or more of a digital image, an audio file, or a video file. In various implementations, the plurality of input parameters may include (e.g., identify) a plurality of digital images. In various implementations, the plurality of input parameters may include a plurality of audio files or video files.

[0009] In various implementations, the one or more object-oriented constructs may include a first object-oriented construct representing a step to be carried out in furtherance of performing the task, and a second object-oriented construct representing a solution to the task. The second object-oriented construct may incorporate one or more instances of the first object-oriented construct.

[0010] In various implementations, the output object comprises one or more of a raster-based graphic or a vector-based graphic. In various implementations, the output object may include one or more of an audio file or a video file.

[0011] Other implementations may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described above. Yet another implementation may include a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described above.

[0012] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.Brief Description of the Drawings

[0013] Fig. 1 schematically depicts an example environment in which disclosed techniques may be employed, in accordance with various implementations.

[0014] Fig. 2 depicts an example input object, desired output type, and exemplar that may be assembled into an input prompt of a generative model, in accordance with various implementations.

[0015] Fig. 3 depicts an example input prompt assembled using the components depicted in Fig. 2.

[0016] Fig. 4 depicts an example generative model output that is predicted when the input prompt of Fig. 2 is processed using a generative model, in accordance with various implementations.

[0017] Fig. 5 schematically depicts an example of how various components of an input prompt assembled in accordance with various implementations described herein may influence and / or constrain generative model output.

[0018] Fig. 6 depicts an example of how techniques described herein may be used to operate on multiple different modalities of data.

[0019] Fig. 7 depicts another example of how techniques described herein may be used, in accordance with various implementations.

[0020] Fig. 8 depicts another example of how techniques described herein may be used, in accordance with various implementations.

[0021] Fig. 9 depicts another example of how techniques described herein may be used, in accordance with various implementations.

[0022] Fig. 10 depicts another example of how techniques described herein may be used, in accordance with various implementations.

[0023] Fig. 11 depicts another example of how techniques described herein may be used, in accordance with various implementations.

[0024] Fig. 12 depicts an example of how techniques described herein may be used to generate non-textual content, in accordance with various implementations.

[0025] Fig. 13 schematically depicts a flowchart demonstrating an example of how techniques described herein may be carried out.

[0026] Fig. 14 schematically depicts an example architecture of a computer system.Detailed Description

[0027] Implementations described herein relate to formulating generative model invocations as object-to-object mappings (extending beyond traditional text-to-text mappings) using object- oriented constructs as schema in generative model input prompts. More particularly, but not exclusively, techniques described herein enable the use of familiar programming object-oriented constructs such as classes, types, and / or objects to prompt (or “condition”) generative models to generate output that is structured and / or consistent. For instance, a user can cause an LLM to predict a solution to a math problem along with a step-by-step resolution by defining a class with steps and answer fields.

[0028] Techniques described herein provide for a variety of technical advantages. Employing object-oriented classes / types as schemas for prompting generative models may in effect establish a Turing-complete language for programming generative models, reducing the need for intricate and / or nuanced prompt engineering using natural language. Moreover, extending the text-to-text mapping capabilities of generative models such as LLMs to object-to-object mappings unifies input and output interfaces across multiple different modalities of data, such as text, imagery, videos, audio, structured data, etc. For example, techniques described herein allow entities represented in different modalities, such as an image of a cat and a textual description of a dog, to be operated upon (e.g., semantically compared) in a single generative model invocation.

[0029] In computer programs, object-oriented constructs such as objects, classes, and / or types serve as runtime entities that represent various concepts, such as people, places, things, data tables, or anything else a computer program uses and / or acts upon. Programming logic outside of these object-oriented constructs essentially involve mapping of the constructs according to rules and / or heuristics.

[0030] In various implementations, a generative model (GAT) invocation may be modeled as a function q that takes, as input, input object(s) a desired output type Z, and optionally, one or more few-shot exemplars E, to generate output object(s) F in accordance with the following equation:GM q : {X, T, E) —> YThe inputs and / or outputs of the function q may be expressed using programming language symbols, e.g., in both the input prompts X, T, E) and the generated responses F.

[0031] An input object X may express a task to be performed using the generative model. A task may come in various forms, such as an analytic task (e.g., “sort these objects by size,” “group these objects into clusters of similar objects”), a mathematical task (e.g., “compute (l+2)x3)”), an image generation task (e.g., “plot y + z = x2in 3D with colors”, “use random fractals to generate a map of a coastline”), and so forth. In some implementations, an input object A may be composed using a programming language, such as the same object-oriented programming language that is used to compose a desired output type T. If the task to be performed using the generative model is “Plan a two-day trip in San Francisco,” an input object X for this task may be defined, for instance, as follows:Tempi ate(“Work out a plan for {{trip}} ”, trip=Trip(destination=“San Francisco”, days=2) )

[0032] This particular example input object X demonstrates how an input object may include multiple input parameters (e.g., key-value pairs), such as “destination^ San Francisco’” and “days=2.” In various implementations, these multiple input parameters may have the same modality or different modalities. For example, one input parameter may be an image (e.g., identified as a uniform resource locator or “URL”) that depicts a first type of animal. Another input parameter may be text that describes or identifies a second type of animal. Yet another input may be an audio file (e.g., identified as a URL) that, if rendered at a speaker, would cause the sound made by a third animal to be output. And so on.

[0033] A desired output type T (also referred to herein as a “schema”) may serve as a blueprint guiding token generation. In some implementations, T may be composed using an object- oriented programming language (e.g., the same language used to compose X). For instance, T may be composed using the object-oriented programming language to include one or more object-oriented constructs (e.g., classes, objects, types) that define structure to impose on generative model output. Essentially, T can be used as a programming language that facilitates execution of generative models such as LLMs. LLMs simulate neural computers by generating tokens in strict accordance with the desired object type T, thereby replicating an ordered execution of instructions dictated by the defined fields of T. Various programming constructs, such as sequences, branches, loops, and recursion, which provide a robust foundation for expressing programs of varying complexities, can be derived from diverse type annotationsembedded within the desired output type T. This endows the desired output type T with Turing completeness, establishing T as a versatile language for programming generative models such as LLMs.

[0034] Exemplars E may be used for few-shot learning to guide the object-to-object mapping of the generative model. In some implementations, an exemplar E may include an exemplar input object XE, an exemplar output type TE, and an exemplar output object YE that is structured in accordance with the exemplar desired output type TE. Providing such an exemplar E may serve multiple purposes. Firstly, it may provide an intuitive illustration of the mapping conventions to be used in the function q. This may facilitate the comprehension and handling of generation details. Secondly, the exemplar E may enable the incorporation of domain-specific knowledge such as programming language knowledge, thereby enhancing the reasoning capability for specific mappings. Through extensive experimentation, it has been observed that even a simple exemplar E such as the " 1 + 1" exemplar shown below may effectively facilitate generation of structured and / or consistent output object(s) Y across diverse applications.INPUT OBJECT:1 + 1 =RESULT TYPE:Answer python class Answer: result: intRESULT OBJECT: python Answer(result=2)

[0035] It may not always be the case that the user is sufficiently fluent in programming to compose input prompts X, T, E) using object-oriented programming language(s). Accordingly, in some implementations, various heuristics and / or machine learning model(s) (e.g., the same generative model as described above or a different “out-of-band” model) may be used to convert a natural language input from the user into a structured input object X similar to those described previously. For example, the user’s natural language request may be enclosed in quotations and stored as a string variable within a larger programming language statement, class, function, etc. In some such implementations, the heuristics / model(s) may also be used to automaticallygenerate a corresponding desired object type T. For example, as users successfully implement techniques described herein to generate output objects F that comport with desired output types , pairs of training examples X T may be used to train and / or fine-tune the out-of-band model to map users’ natural language statements into structured input objects A and desired object types T.

[0036] Once the input prompt X, T, E) is processed using the generative model to predict output object(s) Y structured in accordance with one or more of the object-oriented constructs of the desired output type T, the output object(s) Y may be used for various purposes. In some implementations in which the generative model is part of a larger workflow, one or more downstream processes may be configured to receive input that is structured in accordance with one or more of the object-oriented constructs. By applying techniques described herein, when the output object(s) Y are passed as input(s) to such downstream process of the workflow, the downstream process is able to successfully act upon the output object(s) Y because they are structured as expected.

[0037] Fig. l is a schematic diagram illustrating components that can cooperate to carry out selected aspects of the present disclosure, in accordance with various implementations. The various components depicted in Fig. 1, particularly those components forming a knowledge system 100, may be implemented using any combination of hardware and software. The components of Fig. 1 are depicted as being communicatively coupled with each other via one or more networks 199, which may include one or more personal area networks, local area networks, and / or wide area networks (e.g., the Internet). However, this is not meant to be limiting. Various aspects of the present disclosure that are described as being performed by and / or stored on system 100 can alternatively be performed by and / or stored elsewhere and / or distributed across multiple systems, such as between system 100 and a client device 124.

[0038] In some implementations, knowledge system 100 may include one or more computing devices cooperating to perform selected aspects of the present disclosure. An example of such a computing device is depicted schematically in Fig. 14. In some implementations, knowledge system 100 may include one or more servers forming part of what is often referred to as a “cloud” infrastructure, or simply “the cloud.” Alternatively, one or more components of system 100 may be operated by client device 124.

[0039] Knowledge system 100 may include a prompt generation engine 102 communicatively coupled with one or more generative models 104 trained to generate input prompts, a generative model (GM) response generation engine 106 communicatively coupled with one or more generative models 108 (which in some cases may be the same as 104), and one or more downstream processes 120. Any of elements 102, 104, 106, 108, and / or 120 may be implemented using any combination of hardware and software. Moreover, any of elements 102, 104, 106, 108, and / or 120 may be combined with other(s) of elements 102, 104, 106, 108, and / or 120.

[0040] In various implementations, a user 122 may interact with knowledge system 100 using a client device 124. While depicted as a tablet computer or smart phone in Fig. 1, client device 124 may take other forms, such as a desktop or laptop computer, in-vehicle computing device, augmented reality (AR) and / or virtual reality (VR) headset or glasses, standalone “smart” speakers, etc.

[0041] While shown as separate systems that communicate using network(s) 199, this is not meant to be limiting. Aspects of knowledge system 100 may be implemented in whole or in part on client device 124. If client device 124 includes sufficient computing resources, and / or generative model(s) it uses can be made sufficiently “lean,” it may be desirable to implement techniques described herein locally on client device 124 to avoid latency introduced by a round trip across network(s) 199.

[0042] Techniques described herein related to using object-oriented and / or symbolic prompting. However, it may be the case that user 122 is unfamiliar with object-oriented programming and / or is not inclined to write code in order to generate content using one or more generative models 108. Accordingly, in some implementations, user 122 may operate client device 124 to issue a natural language request 110 to be provided to knowledge system 100. User 122 may type the natural language request 110 or may speak it aloud, and the utterance may be recorded and transcribed by an automated speech recognition (ASR) component (not depicted) hosted by client device 124 and / or by knowledge system 100.

[0043] Natural language request 110 (in textual and / or embedding form) may be processed by prompt generation engine 102 using one or more generative models 104 trained to map natural language snippets to input object(s) 112 and / or desired output type(s) 114. One or both of input object(s) 112 and / or desired output types 114 may include object-oriented construct(s) that arecomposed using an object-oriented programming language, such as C++, Java, Ada, Python, Lisp, C#, Perl, PHP, Ruby, etc. These object-oriented constructs (e.g., classes, types, and / or objects) may define structure to impose on generative model output predicted by GM response generation engine 106 using one or more generative models 108. Examples of input object(s) 112 and desired output type(s) 114 will be provided herein.

[0044] In other implementations in which user 122 has sufficient knowledge / expertise to manually create object-oriented input prompts, user 122 may compose input object(s) 112 and / or desired output type(s) 114 manually. In some such implementations, prompt generation engine 102 and / or generative model(s) 104 may be omitted, or not used. And where applicable, user 122 may also manually compose one or more exemplars 116. Also, in some implementations, the functionality attributed herein to prompt generation engine 102 may be incorporated into GM response generation engine 106, and / or generative models 104 and 108 may be the same.

[0045] Whether composed by user 122 or generated automatically, data indicative of input object(s) 112 and desired output type(s) 114 may be processed by GM response generation engine 106 using one or more generative models 108. In some implementations, data 112 and 114 may be processed by GM response generation engine 106 in combination with one or more exemplars 116 that demonstrate example mapping(s) from reference input object(s) and desired output type(s). Based on this processing, GM response generation engine 106 may predict one or more output objects 118. Output object(s) 118 may be in a structured and / or predictable form based at least on the desired output type(s) 114 used by GM response generation engine 106 to prompt generative model(s) 108.

[0046] In various implementations, knowledge system 100 may host one or more downstream processes 120 configured to receive input that is structured in accordance with various schema, e.g., such as one or more object-oriented constructs contained in desired output type(s) 114. As an example, one downstream process may be configured to process a structured list of integers. If output object 118 generated by GM response generation engine using generative model(s) 108 were not constrained as described herein, output object 118 might express a list of numbers numerally, in word form, as floats, etc. By practicing selected aspects of the present disclosure, it is possible to constrain output object(s) 118 generated by GM response generation engine 106 to form(s) that are predictable and / or capable of processing by downstream process(es) 120.While downstream processes 120 are depicted as part of knowledge system 100, this is not required. These processes may be distributed across other computer systems that are separate from those hosting knowledge system 100.

[0047] Figs. 2-4 depict a non-limiting example of how techniques described herein may be used to plan a trip itinerary. Fig. 2 depicts an input object 212, a desired output type 214, and a single exemplar 216 — these components may share characteristics with components 112-116 in Fig. 1. Input object 212 defines a “template” class / object that includes, as a string variable, a natural language snippet, “Work out a plan for {{trip}}.” Input object 212 also defines an instance “trip” of a “Trip” object that itself includes a destination string (“San Francisco”) and variable “days” that is set to “2.” In sum, input object 212 is a structured request, e.g., from user 122, to have GM response generation engine 106 use one or more generative models 108 to generate a two-day itinerary to San Francisco.

[0048] Desired output type 214 defines two classes: an Activity and an Itinerary. The Activity class defines a “place of interesf ’ string variable and a “description” string variable. The Itinerary class defines an integer variable called “day” and a list called “activities” that includes one or more instances of the Activity class. No methods or functions are defined as part of these classes, nor do they need to be for implementations described herein to function properly.

[0049] While not required, exemplar 216 is included to provide additional conditioning for generative model(s) 108. Here, exemplar 216 includes a “MappingExample” that maps an input of “Compute 1+1” to an output of “ Answer(result=2), based on the schema (e.g., desired output type 114) of “Answer.” Notably, the only similarity between exemplar 216 and input object 212 or desired output type 214 is that all three are composed in the same object-oriented programming language. The functionality of exemplar 216 is unrelated to the desired functionality of input object 212. Nonetheless, even such a simple mapping is effective in guiding (further conditioning) output object(s) predicted by GM response generation engine 106 using generative model(s) 108. In some implementations, simple exemplars such as 214 of Fig. 2 are used as “default” exemplars, e.g., where user 122 fails to provide exemplar(s).

[0050] Fig. 3 depicts an input prompt 330 that may be assembled using input object 212, desired output type (schema) 214, and exemplar 216 in Fig. 2. Input prompt 330 includes a request 332, “Please response to INPUT OBJECT with RESULT OBJECT based on RESULT TYPE .” Request 332 may be composed by user 122 or composed automatically, e.g., without requiringuser 122 to do so. For example, request 332 may be assembled into input prompt 330 in response to user 122 composing input object 212, desired output type 214, and / or exemplar 216, or in response to components 212-216 being predicted by prompt generation engine 102 based on natural language input of user 122.

[0051] Input prompt 330 also includes content 316A-C that corresponds to exemplar 216, including an exemplar input object 316A (“INPUT OBJECT”), an exemplar desired output type (or schema) 316B (“RESULT TYPE”), and an exemplar target output object 316C(“RESUL T OBJECT”). Thus, it can be seen that content 316A-C provides an example of how a RESULT OBJECT can be generated based on an INPUT OBJECT based on a RESULT TYPE.

[0052] Additionally, input prompt 330 includes an input object 312 corresponding to input object 212 in Fig. 2, as well as a desired output type 314 that corresponds to desired output type 214 in Fig. 2. Finally, input prompt includes a placeholder 334 for a result object to be predicted from input object 312 based on desired output type 314.

[0053] Fig. 4 depicts an example of generative model output 436 that may be generated by GM response generation engine 106 using generative model(s) 108 based on the input prompt 330 of Fig. 3. Generative model output 436 is composed in the same object-oriented language (Python) as were elements 212-216 and 312-316, although this is not necessarily required in all cases. Notably, generative model output 436 is structured in accordance with desired output type 314 of Fig. 3, particularly in accordance with the “Activity” and “Itinerary” classes defined therein. In this example, the first day of a two-day itinerary includes recommended visits to two places of interest (“place of interest” variable of desired output type 316): the Golden Gate Bridge and Fisherman’s Wharf. Also included are a brief description (“description” variable of desired output type 316) of each place of interest. A second day of the itinerary includes recommended visits to Alcatraz Island and Historic Chinatown, as well as descriptions of each.

[0054] Fig. 5 schematically depicts, on the left, an input prompt, and on the right, a response generated by GM response generation engine 106 using generative model(s) 108 on the input prompt. The arrows demonstrate how attention mechanisms come into play when using techniques described herein to constrain and / or impose structure onto generative model output. Symbolic attentions are represented by solid lines. Symbolic attentions serve to refine the generated text on the right by aligning it with the desired output type (e.g. , the schema), therebynarrowing the range of choices at the generation of each output token. Semantic attentions, by contrast, are represented by the dashed arrows. Semantic attentions contribute to the generation of text with meaningful values by focusing on semantic relationships.

[0055] Fig. 6 depicts another example in which techniques described herein are applied, in this instance to facilitate comparison of and / or interaction with multiple different input parameters in multiple different modalities. The variable “image roof ’ is populated with a prefix of a URL path that leads to an inventory of digital images. Anything past this prefix identifies folders and / or files further down the path. The array “files” is populated with four different strings, each identifying a different digital image, including an “African Bush Elephant,” a “Rabbit in Montana,” a “Domestic Cat,” and an “Orangutan.” The variable “images” is populated by concatenating each string of the “files” array with the image root, thereby resulting in four complete paths to images, each path forming a respective input parameter.

[0056] Next, as a desired output type 614, two additional input parameters are defined as two additional classes: a first class called “Whale” and a second class called “Ant.” Notably, each of these additional input parameters / classes is merely defined with a “pass” statement, and without any other variables. In this instance, it is the semantic meaning of the class names “Whale” and “Ant” that will be given semantic attention towards the resulting output.

[0057] Next, a call (“If.query”) is made to apply a generative model (“GeminiPro Vision”) to data. The call includes an input object (“prompt”) of “Name the objects and sort by size: {{objects}}” which will seek out a variable / element called “objects.” The call also includes a “schema” e.g., a desired output type) of “list[str]”. The call further defines the aforementioned “objects” variable / element as including the image paths stored in the “images” variable, plus the Ant and Whale classes. Finally, the call invokes the generative model (GeminiProVision) itself.

[0058] Example generative model output is provided at bottom. The four organisms defined using URL paths (Elephant, Rabbit, Domestic Cat, Orangutan) and the two organisms defined using classes (Whale and Ant) have been shuffled together, correctly sorted by size, and stored in an array, as requested, in spite of the fact that the first four animals are processed by the generative model (e.g., 108) as images and the last two animals are processed based on their names alone. More generally, techniques described herein can be used to cause generative models to reason across modalities. While images and text are used as input parameters in Fig.6, this is not meant to be limiting. Any combination of text, images, videos, audio files, and / orother types of digital content can be intermixed and / or acted upon using techniques described herein.

[0059] Techniques described herein may be used to implement advanced reasoning with an understanding of topology and / or abstractions. An example of how mathematical expressions can be mapped to a tree representation is depicted schematically in Fig. 7. An input prompt 730 is depicted at left and a resulting generative model output 736 is depicted at right. A desired output type is defined using four object-oriented classes: Node, Number (which takes a Node as input), BinaryOp (also takes a Node as input), and Multiply (which takes a BinaryOp as input). The BinaryOp object-oriented class defines left and right Nodes.

[0060] The variable “node” is populated with generative model output. The input “prompt” is set to “1 + 3 * (4 + 2).” The schema (e.g., desired output type) is set to the Node object-oriented class. At right in the resulting generative model output 736, object-oriented code is displayed that reasons through the order of operations required to solve the equation provided as the “prompt” variable. First the numbers 4 and 2 are added (6), then the sum is multiplied by the number 3 (18), and the product is added to the number 1, reaching the final result of 19.

[0061] Techniques described herein may be applicable to various types of generative model prompting techniques, such as chain-of-thought, least-to-most, and / or trial-and-error. Fig. 8 depicts an input prompt 830 at top and a resulting generative model response 836 at bottom. The input prompt includes a natural language question:Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins every day with four. She sells the remainder at the farmer? s market daily for $2 per fresh duck egg. How much does she make every day at the farmer? s market?

[0062] A desired output type is defined to include two object-oriented classes: Step and Solution. As demonstrated in Fig. 5, the “Step” class serves as a symbolic attention. By including “Step” as a class, and by incorporating a field “steps” prior to a “final answer” field of the Solution class, chain-of-thought reasoning is imposed on the resulting generative model output 836. This is borne out in resulting generative model output 836, where the solution is rendered as an ordered sequence of steps, with the “step outpuf ’ field including a current tally of how many eggs Janet has remaining at each step.

[0063] Fig. 9 depicts an example in which trial-and-error prompting is enhanced with techniques described herein. The question to be answered using the generative model is “Please insert +, -,*, / , (, ) around the 4 numbers (1, 2, 3, 4) to produce a valid expression that results in 24.” A desired output type is defined to include two object-oriented classes: Attempt and Solution. The Attempt class includes a string called “expression,” a list called “steps,” and a Boolean called “result_equal_to_24.” The latter variable enhances the attention of the generative model(s) to the outcome of each attempt. Under this schema, generative models may function as loops, attempting various iterations until the termination condition is satisfied.

[0064] The resulting solution provided in responsive generative model output 936 includes first and second attempts to solve the posed question. The first attempt (0) proposed a candidate expression of “1 * 3 * 2 * 3,” which yielded, in three steps, the final result of 18. Because 18 is not equal to 24, the result_equal_to_24 Boolean was expressed as False. The second attempt (1) proposed a candidate expression “(1+3) * (2 * 3),’ which yielded, in three steps, the final result of 24. Because 24 is equal to 24, the result_equal_to_24 Boolean was expressed as True.

[0065] Fig. 10 depicts another example of how programming and generative models can be integrated. In Fig. 10, the question to be answered is: “It takes Amy 4 minutes to climb to the top of a slide. It takes her one minute to slide down. The water slide closes in 15 minutes. How many times can she slide before it closes?” A desired object type is defined using the object- oriented class Solution that includes an “answer” integer field. Two calls are then made to the generative model. In the first call, the “question” is provided as the input object and the “Solution” class is provided as the desired output type. The generative model is called, e.g., with a temperature of 0.7, for five iterations, to populate a “samples” array. The result (samples) is then printed and includes five individual samples: “[3, 3, 11, 3, 3].” Next, the generative model is called again, this time with the input prompt, “Majority vote from {{samples}}”. The “{{samples}}” field is populated with the values from the “samples” array, and the “Majority vote” language is then processed by the generative model to generate the output of “3”, since the majority of the values in the “samples” array were the value 3.

[0066] Fig. 11 depicts an example of how an input object and desired output type can be used to define a function’s signature. From the perspective of user 122, the signatures of such functions may be indistinguishable from “regular” (i.e. rules based) functions. However, some functions created using techniques described herein may operate on fuzzy logic, as opposed to rules-based logic.

[0067] Fig. 11 depicts an input prompt 1130 and responsive generative model output 1136. In input prompt 1130, a desired output type is defined as including the object-oriented classes (or objects) “Code” and “Cluster.” The “Code” class includes a single string element “snippet” and the “Cluster” class which includes two elements: a “fields” array of Code classes and a string called “reason.”

[0068] A generative model call includes, as an input object, the statement, “Clustering {{objects}},” as well as a desired output type in the form of a list of “Cluster” objects. The “objects” field referenced in the input object is then defined as an array that includes the following Code elements (written in Python): ‘x = y + 1’, ‘print(“hello world”)’, ‘foo(l,2)’, ‘print(“hello universe”)’, and ‘x = y - 1’. When this input prompt is processed by GM response generation engine 106 using generative model(s) 108 (e.g., GeminiPro), the resulting generative model output 1136 is structured as a list of Clusters of semantically similar Code elements. For example, “x = y + 1” and “x = y - 1” are clustered together based on the reason, “Arithmetic operations on x and y.” Similarly, “print(“hello world”)” and “print(“hello universe”)” are clustered together based on them both being print statements.

[0069] Fig. 12 demonstrates an example of how techniques described herein may be used to predict generative model output that is itself structured and / or that is operable to generate additional structured content. More particularly, Fig. 12 demonstrates how techniques described herein are not limited to generating structured textual output (e.g., code), as in the previous examples. In this example, the ultimate output 1240, which may be generative model output itself or generated using generative model output (e.g., Python code that can be rendered), takes the form of imagery.

[0070] In the input prompt 1230, a desired output type is defined to include an object-oriented class (or object) called “PythonCode” that includes a single string element called “source.” Next, a self-defining function call is invoked. Then, the generative model is invoked with the input object “Plot z = x2+y2in 3D” and the class PythonCode as the desired output type. This results in the generation of Python code (not depicted) that, when rendered, causes the ultimate output 1240 to include a 3D plot of the requested equation. In various implementations, the 3D plot (or any imagery created using techniques described herein may be generated as raster-based graphics or vector-based graphics. Moreover, techniques described herein can be used togenerate other modalities of data, including but not limited to audio content, videos, animations, various types of documents (e.g., emails), etc.

[0071] Referring now to Fig. 13, an example method 1300 of practicing selected aspects of the present disclosure is described. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig. 1. Moreover, while operations of method 1300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

[0072] At optional block 1302, the system, e.g., by way of prompt generation engine 102, may process a natural language snippet received (e.g., typed, or spoken and transcribed) from user 122 using one or more generative models (e.g., 104, 108) to generate an input object (e.g., 112, 212, 312) and a desired output type (e.g., 114, 214, 314, 614). In various implementations, the desired output type may include one or more object-oriented constructs that are composed using an object-oriented programming language, and that that define structure to impose on generative model output predicted using the generative model. As mentioned previously, the operations of block 1302 may be implemented where user 122 is either unable or unwilling to compose object-oriented content for an input prompt themselves. To this end, the generative model(s) 104 used as part of block 1302, which may be the same as 108 in some implementations, may be trained and / or fine-tuned to map natural language snippets to input objects and / or desired output types that are composed using object-oriented programming languages.

[0073] Whether the input object and desired output type are generated at block 1302 or manually composed by user 122, at block 1304, the system may assemble an input prompt for one or more of the generative models. In various implementations, the input prompt may be assembled to include the input object (e.g., 112, 312) and the desired output type (e.g., 114, 314, 614). At optional block 1306, the system may assemble one or more exemplars (e.g., 116, 316) into the input prompt. These exemplars may be provided by user 122, generated using generative model(s) 104 / 108, and / or provided automatically and / or implicitly. For instance, the “Compute 1 + 1” mapping exemplar 216 / 316 in Figs. 2-3 has been shown to be applicable and effective across diverse applications.

[0074] At block 1308, the system, e.g., by way of GM response generation engine 106, may process the input prompt using one or more of the generative models (108) to predict an outputobject (e.g., 118, 436, 736, 836, 936, 1036, 1136, 1236). In various implementations, the output object may be structured in accordance with one or more of the object-oriented constructs included in the desired output type.

[0075] In some implementations, at block 1310, the system may pass the output object (e.g., 118) to one or more downstream processes (e.g., 120), e.g., of a larger workflow of which GM response generation engine 106 and generative model(s) 108 are merely a part. The downstream process(es) may be configured to receive input or otherwise process data that is structured in accordance with one or more of the object-oriented constructs included in the desired output type (e.g., 114, 214, 314, 614).

[0076] Fig. 14 is a block diagram of an example computer system 1410. Computer system 1410 typically includes at least one processor 1414 which communicates with a number of peripheral devices via bus subsystem 1412. These peripheral devices may include a storage subsystem 1424, including, for example, a memory subsystem 1425 and a file storage subsystem 1426, user interface output devices 1420, user interface input devices 1422, and a network interface subsystem 1416. The input and output devices allow user interaction with computer system 1410. Network interface subsystem 1416 provides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

[0077] User interface input devices 1422 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computer system 1410 or onto a communication network.

[0078] User interface output devices 1420 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from computer system 1410 to the user or to another machine or computer system.

[0079] Storage subsystem 1424 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 1424 may include the logic to perform selected aspects of method 700, and / or to implement one or more aspects of the various components depicted in Fig. 1. Memory 1425 used in the storage subsystem 1424 can include a number of memories including a main random-access memory (RAM) 1430 for storage of instructions and data during program execution and a read only memory (ROM) 1432 in which fixed instructions are stored. A file storage subsystem 1426 can provide persistent storage for program and data files, and may include a hard disk drive, a CD-ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored by file storage subsystem 1426 in the storage subsystem 1424, or in other machines accessible by the processor(s) 1414.

[0080] Bus subsystem 1412 provides a mechanism for letting the various components and subsystems of computer system 1410 communicate with each other as intended. Although bus subsystem 1412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0081] Computer system 1410 can be of varying types including a workstation, server, computing cluster, blade server, server farm, smart phone, smart watch, smart glasses, set top box, tablet computer, laptop, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 1410 depicted in Fig. 14 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system 1410 are possible having more or fewer components than the computer system depicted in Fig. 14.

[0082] While several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

CLAIMSWhat is claimed is:

1. A method implemented using one or more processors and comprising: assembling an input prompt for a generative model, the input prompt comprising an input object and a desired output type, wherein the input object expresses a task to be performed using the generative model, and wherein the desired output type comprises one or more object-oriented constructs, composed using an object-oriented programming language, that define structure to impose on generative model output predicted using the generative model; and processing the input prompt using the generative model to predict an output object, wherein the output object is structured in accordance with one or more of the object-oriented constructs.

2. The method of claim 1, wherein the input object expresses the task to be performed using natural language.

3. The method of claim 1 or 2, wherein the input object expresses the task to be performed using the object-oriented programming language.

4. The method of any of the preceding claims, wherein the task to be performed comprises an analytic task.

5. The method of any of the preceding claims, wherein the task to be performed comprises a mathematical task.

6. The method of any of the preceding claims, further comprising passing the output object as input to a downstream process, wherein the downstream process is configured to receive input that is structured in accordance with one or more of the object-oriented constructs.

7. The method of any of the preceding claims, wherein the input prompt further comprises at least one exemplar, wherein the at least one exemplar comprises an exemplar input object, an exemplar output type, and an exemplar output object that is structured in accordance with the exemplar desired output type.

8. The method of claim 7, wherein the exemplar output type comprises one or more additional object-oriented constructs composed using the object-oriented programming language.

9. The method of any of the preceding claims, further comprising processing data indicative of a natural language snippet that expresses the task using the same generative model or a different generative model to predict the input object and the desired output type.

10. The method of any of the preceding claims, wherein the input object comprises one or more statements, composed in the object-oriented programming language, that set a value of a string variable to a natural language snippet that expresses the task.

11. The method of claim 10, further comprising training the same generative model or a different model based on the natural language snippet and the desired output type.

12. The method of any of the preceding claims, wherein the input object comprises a plurality of input parameters for performance of the task.

13. The method of claim 12, wherein the plurality of input parameters comprise a first input parameter in a first modality and a second input parameter in a second modality that is different than the first modality.

14. The method of claim 13, wherein the first modality comprises structured text or natural language.

15. The method of claim 14, wherein the second modality comprises a digital image.

16. The method of claim 14, wherein the second modality comprises an audio file.

17. The method of claim 14, wherein the second modality comprises a video file.

18. The method of any of claims 12-17, wherein the plurality of input parameters comprise a plurality of digital images.

19. The method of any of claims 12-18, wherein the plurality of input parameters comprise a plurality of audio files or video files.

20. The method of any of the preceding claims, wherein the one or more object- oriented constructs comprise a first object-oriented construct representing a step to be carried out in furtherance of performing the task, and a second object-oriented construct representing a solution to the task, wherein the second object-oriented construct incorporates one or more instances of the first object-oriented construct.

21. The method of any of the preceding claims, wherein the output object comprises one or more of a raster-based graphic or a vector-based graphic.

22. The method of any of the preceding claims, wherein the output object comprises one or more of an audio file or a video file.

23. A system comprising one or more processors and memory storing instructions that cause the one or more processors to perform any of the methods of claims 1-22.

24. At least one non-transitory computer-readable medium comprising instructions that, in response execution by one or more processors, cause the one or more processors to perform any of the methods of claims 1-22.

Images