System, method, and computer program product for integrated processing of generative and instructive prompts in machine learning models

Abstract

Systems, methods, and computer program products are provided for integrated processing of generative and instructive prompts in machine learning models. An example system includes a processor configured to receive reference data, store a representation of the reference data, and receive a prompt. The processor is also configured to determine a first portion of the prompt associated with a generative prompt and a second portion of the prompt associated with an executable action. The processor is further configured to retrieve a subset of the representation and determine a generative output from a machine learning model based on the subset and the first portion of the prompt. The processor is further configured to generate content based on the generative output, determine an encoding of a plurality of action steps, and execute the executable action using a sequence-to-sequence decoder model and based on the content and the encoding.

Description

SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR INTEGRATED PROCESSING OF GENERATIVE AND INSTRUCTIVE PROMPTS IN MACHINE LEARNING MODELSBACKGROUND1 . Technical Field

[0001] This disclosure relates generally to applied machine learning systems and, in non-limiting embodiments or aspects, to systems, methods, and computer program products for integrated processing of generative and instructive prompts in machine learning models.2. Technical Considerations

[0002] Retrieval augmented generation (RAG) models represent an area of technical improvement for generative machine learning models, by permitting generative machine learning models to augment the creation of generative output with reference to contextual datasets (e.g., data stored in memory to provide reference data for generative output). However, RAG models have a number of technical limitations that may restrict their accuracy. For example, RAG models may have increased latency, both in the time required to retrieve data from contextual datasets, and in the recency of contextual datasets, owing to historic-based data consumption (e.g., compared to real-time data sources). This latency may result in slower and / or less accurate model output, which may reduce the efficiency of any downstream processes reliant on the model output, or may delay model output if the accuracy criterion is instead prioritized.

[0003] Moreover, to make RAG models more accurate, it may be necessary to store more diverse and extensive contextual datasets, which may require more memory and / or bandwidth for use of the contextual datasets. This may also result in slower output times, which reduces the efficiency of the overall modeling system. As such, scalability is an existing technical problem to be addressed in RAG modeling systems. Further to that end, as a contextual dataset becomes larger, it may become less likely for a RAG model to identify connections between disparate data references in the contextual dataset. If the retrieval component of a RAG model fails to fetch the most relevant contextual data, the generative output may be suboptimal or inaccurate,which may deteriorate the utility of downstream processes relying on the generative machine learning model.

[0004] Beyond limitations related to data storage, communication speed, and dataset complexity, RAG models may be limited by data modality (e.g., text, image, video, audio, etc.). Contextual data may be oriented to one modality, with the expectation that queries will be directed to the same modality. This may functionally limit the technical usefulness of the RAG model if different modalities — both in contextual data and in expected generative output — cannot be handled. Furthermore, RAG models may be further limited in functionality if prompts to the RAG models seek actions to be executed and not just output to be generated.

[0005] There is a need in the art for a technically improved RAG modeling system, which is configured to reduce latency, improve scalability, optimize the storage of data in memory, improve the accuracy of generated output, manage multiple data modalities, and combine generative output with automatically executed actions.SUMMARY

[0006] Accordingly, provided are improved systems, methods, and computer program products for integrated processing of generative and instructive prompts in machine learning models.

[0007] According to non-limiting embodiments or aspects, provided is a system for integrated processing of generative and instructive prompts in machine learning models. The system includes at least one processor configured to receive reference data, store a representation of the reference data in a datastore, and receive a prompt. The at least one processor is also configured to determine a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action. The at least one processor is further configured to retrieve a subset of the representation of the reference data from the datastore based on the first portion of the prompt. The at least one processor is further configured to determine a generative output from a machine learning model based on an input including (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt. The at least one processor is further configured to generate content responsive to the first portion of the prompt based on the generative output. The at least one processor is further configured to determine an encoding of a plurality of action steps based on the second portion of the prompt.The at least one processor is further configured to execute the at least one executable action using a sequence-to-sequence decoder model and based on the content and the encoding of the plurality of action steps.

[0008] In some non-limiting embodiments or aspects, the reference data may include batch data and stream data. When receiving the reference data, the at least one processor may be further configured to receive the batch data and the stream data in separate threads of a multi-threaded communication channel.

[0009] In some non-limiting embodiments or aspects, the at least one processor may be further configured to, in response to receiving the reference data, pre-process the reference data to conform to a predetermined data format, to produce formatted reference data, and dispatch the formatted reference data to at least one queue.

[0010] In some non-limiting embodiments or aspects, the at least one processor may be further configured to, in response to the formatted reference data being dispatched to the at least one queue, perform a series of steps. The series of steps may include retrieving, in queue order, the formatted reference data from the at least one queue. The series of steps may also include generating a plurality of tokens based on the formatted reference data. The series of steps may further include generating a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups. The series of steps may further include converting the plurality of token groups into a plurality of representations using a distributed embedding model.

[0011] In some non-limiting embodiments or aspects, the representation of the reference data may include the plurality of representations. When storing the representation of the reference data, the at least one processor may be further configured to store the plurality of representations in the datastore.

[0012] In some non-limiting embodiments or aspects, the distributed embedding model may be executed at least partly on a server cluster including at least one primary node and a plurality of secondary nodes. When converting the plurality of token groups into the plurality of representations using the distributed embedding model, the at least one processor may be further configured to, for each token group of the plurality of token groups, perform a series of steps. The series of steps may include transmitting the token group to the primary node. The series of steps may also include determining a corresponding secondary node of the plurality of secondary nodesbased at least partly on a data modality of the token group. The series of steps may further include directing the token group to the corresponding secondary node from the primary node. The series of steps may further include generating a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

[0013] In some non-limiting embodiments or aspects, each secondary node of the plurality of secondary nodes may include a complete set of layers of the distributed embedding model. When determining the corresponding secondary node, the at least one processor may be further configured to determine the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

[0014] In some non-limiting embodiments or aspects, when determining the encoding of the plurality of action steps based on the second portion of the prompt, the at least one processor may be further configured to convert the second portion of the prompt into a vector representation using a transformer encoder model, and determine the encoding of the plurality of action steps using a sequence-to-sequence encoder model and based on the vector representation.

[0015] In some non-limiting embodiments or aspects, when receiving the prompt, the at least one processor may be further configured to receive the prompt via a user interface of a computing device of a user. The at least one processor may be further configured to extract a plurality of features based on the user interface using a neural network model. The plurality of features may be associated with (i) a plurality of elements of the user interface and (ii) a plurality of relationships between the plurality of elements. The at least one processor may be further configured to construct a graph of the user interface based on the plurality of features. The graph may include a plurality of nodes and a plurality of edges, each node of the plurality of nodes associated with an element of the plurality of elements, and each edge of the plurality of edges associated with a relationship of the plurality of relationships.

[0016] In some non-limiting embodiments or aspects, when executing the at least one executable action, the at least one processor may be further configured to traverse the graph of the user interface using the sequence-to-sequence decoder model and based on the encoding of the plurality of action steps.

[0017] According to non-limiting embodiments or aspects, provided is a computer- implemented method for integrated processing of generative and instructive promptsin machine learning models. The method includes receiving, with at least one processor, reference data. The method also includes storing, with at least one processor, a representation of the reference data in a datastore. The method further includes receiving, with at least one processor, a prompt. The method further includes determining, with at least one processor, a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action. The method further includes retrieving, with at least one processor, a subset of the representation of the reference data from the datastore based on the first portion of the prompt. The method further includes determining, with at least one processor, a generative output from a machine learning model based on an input including (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt. The method further includes generating, with at least one processor, content responsive to the first portion of the prompt based on the generative output. The method further includes determining, with at least one processor, an encoding of a plurality of action steps based on the second portion of the prompt. The method further includes executing, with at least one processor, the at least one executable action using a sequence-to-sequence decoder model and based on the content and the encoding of the plurality of action steps.

[0018] In some non-limiting embodiments or aspects, the method may also include, in response to receiving the reference data, pre-processing, with at least one processor, the reference data to conform to a predetermined data format, to produce formatted reference data. The method may further include, in response to receiving the reference data, dispatching, with at least one processor, the formatted reference data to at least one queue.

[0019] In some non-limiting embodiments or aspects, the method may also include, in response to the formatted reference data being dispatched to the at least one queue, a series of steps. The series of steps may include retrieving, with at least one processor and in queue order, the formatted reference data from the at least one queue. The series of steps may also include generating, with at least one processor, a plurality of tokens based on the formatted reference data. The series of steps may further include generating, with at least one processor, a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups. The series of steps may further include converting, with at least one processor, theplurality of token groups into a plurality of representations using a distributed embedding model.

[0020] In some non-limiting embodiments or aspects, the distributed embedding model may be executed at least partly on a server cluster including at least one primary node and a plurality of secondary nodes. Converting the plurality of token groups into the plurality of representations using the distributed embedding model may further include, for each token group of the plurality of token groups, a series of steps. The series of steps may include transmitting the token group to the primary node. The series of steps may also include determining a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group. The series of steps may further include directing the token group to the corresponding secondary node from the primary node. The series of steps may further include generating a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

[0021] In some non-limiting embodiments or aspects, each secondary node of the plurality of secondary nodes may include a complete set of layers of the distributed embedding model. Determining the corresponding secondary node may further include determining the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

[0022] According to non-limiting embodiments or aspects, provided is a computer program product for integrated processing of generative and instructive prompts in machine learning models. The computer program product includes at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to receive reference data, store a representation of the reference data in a datastore, and receive a prompt. The program instructions also cause the at least one processor to determine a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action. The program instructions further cause the at least one processor to retrieve a subset of the representation of the reference data from the datastore based on the first portion of the prompt. The program instructions further cause the at least one processor to determine a generative output from a machine learning model based on an input including (i) the subset of the representation of the reference data and (ii) at least partof the first portion of the prompt. The program instructions further cause the at least one processor to generate content responsive to the first portion of the prompt based on the generative output. The program instructions further cause the at least one processor to determine an encoding of a plurality of action steps based on the second portion of the prompt. The program instructions further cause the at least one processor to execute the at least one executable action using a sequence-to- sequence decoder model and based on the content and the encoding of the plurality of action steps.

[0023] In some non-limiting embodiments or aspects, the program instructions may further cause the at least one processor to, in response to receiving the reference data, pre-process the reference data to conform to a predetermined data format, to produce formatted reference data, and dispatch the formatted reference data to at least one queue.

[0024] In some non-limiting embodiments or aspects, the program instructions may further cause the at least one processor to, in response to the formatted reference data being dispatched to the at least one queue, perform a series of steps. The series of steps may include retrieving, in queue order, the formatted reference data from the at least one queue. The series of steps may also include generating a plurality of tokens based on the formatted reference data. The series of steps may further include generating a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups. The series of steps may further include converting the plurality of token groups into a plurality of representations using a distributed embedding model.

[0025] In some non-limiting embodiments or aspects, the distributed embedding model may be executed at least partly on a server cluster including at least one primary node and a plurality of secondary nodes. The program instructions that cause the at least one processor to convert the plurality of token groups into the plurality of representations using the distributed embedding model may further cause the at least one processor to, for each token group of the plurality of token groups, perform a series of steps. The series of steps may include transmitting the token group to the primary node. The series of steps may also include determining a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group. The series of steps may further include directing the token group tothe corresponding secondary node from the primary node. The series of steps may further include generating a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

[0026] In some non-limiting embodiments or aspects, each secondary node of the plurality of secondary nodes may include a complete set of layers of the distributed embedding model. The program instructions that cause the at least one processor to determine the corresponding secondary node may further cause the at least one processor to determine the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

[0027] Further non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0028] Clause 1 : A system comprising: at least one processor configured to: receive reference data; store a representation of the reference data in a datastore; receive a prompt; determine a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action; retrieve a subset of the representation of the reference data from the datastore based on the first portion of the prompt; determine a generative output from a machine learning model based on an input comprising (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt; generate content responsive to the first portion of the prompt based on the generative output; determine an encoding of a plurality of action steps based on the second portion of the prompt; and execute the at least one executable action using a sequence-to-sequence decoder model and based on the content and the encoding of the plurality of action steps.

[0029] Clause 2: The system of clause 1 , wherein the reference data comprises batch data and stream data, and wherein, when receiving the reference data, the at least one processor is further configured to receive the batch data and the stream data in separate threads of a multi-threaded communication channel.

[0030] Clause 3: The system of clause 1 or clause 2, wherein the at least one processor is further configured to, in response to receiving the reference data: pre- process the reference data to conform to a predetermined data format, to produce formatted reference data; and dispatch the formatted reference data to at least one queue.

[0031] Clause 4: The system of any of clauses 1 -3, wherein the at least one processor is further configured to, in response to the formatted reference data being dispatched to the at least one queue: retrieve, in queue order, the formatted reference data from the at least one queue; generate a plurality of tokens based on the formatted reference data; generate a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups; and convert the plurality of token groups into a plurality of representations using a distributed embedding model.

[0032] Clause 5: The system of any of clauses 1 -4, wherein the representation of the reference data comprises the plurality of representations, and wherein, when storing the representation of the reference data, the at least one processor is further configured to store the plurality of representations in the datastore.

[0033] Clause 6: The system of any of clauses 1 -5, wherein the distributed embedding model is executed at least partly on a server cluster comprising at least one primary node and a plurality of secondary nodes, and wherein, when converting the plurality of token groups into the plurality of representations using the distributed embedding model, the at least one processor is further configured to, for each token group of the plurality of token groups: transmit the token group to the primary node; determine a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group; direct the token group to the corresponding secondary node from the primary node; and generate a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

[0034] Clause 7: The system of any of clauses 1 -6, wherein each secondary node of the plurality of secondary nodes comprises a complete set of layers of the distributed embedding model, and wherein, when determining the corresponding secondary node, the at least one processor is further configured to determine the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

[0035] Clause 8: The system of any of clauses 1 -7, wherein, when determining the encoding of the plurality of action steps based on the second portion of the prompt, the at least one processor is further configured to: convert the second portion of the prompt into a vector representation using a transformer encoder model; and determinethe encoding of the plurality of action steps using a sequence-to-sequence encoder model and based on the vector representation.

[0036] Clause 9: The system of any of clauses 1 -8, wherein, when receiving the prompt, the at least one processor is further configured to receive the prompt via a user interface of a computing device of a user; and wherein the at least one processor is further configured to: extract a plurality of features based on the user interface using a neural network model, the plurality of features associated with (i) a plurality of elements of the user interface and (ii) a plurality of relationships between the plurality of elements; and construct a graph of the user interface based on the plurality of features, the graph comprising a plurality of nodes and a plurality of edges, each node of the plurality of nodes associated with an element of the plurality of elements, and each edge of the plurality of edges associated with a relationship of the plurality of relationships.

[0037] Clause 10: The system of any of clauses 1 -9, wherein, when executing the at least one executable action, the at least one processor is further configured to traverse the graph of the user interface using the sequence-to-sequence decoder model and based on the encoding of the plurality of action steps.

[0038] Clause 11 : A computer-implemented method comprising: receiving, with at least one processor, reference data; storing, with at least one processor, a representation of the reference data in a datastore; receiving, with at least one processor, a prompt; determining, with at least one processor, a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action; retrieving, with at least one processor, a subset of the representation of the reference data from the datastore based on the first portion of the prompt; determining, with at least one processor, a generative output from a machine learning model based on an input comprising (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt; generating, with at least one processor, content responsive to the first portion of the prompt based on the generative output; determining, with at least one processor, an encoding of a plurality of action steps based on the second portion of the prompt; and executing, with at least one processor, the at least one executable action using a sequence-to-sequence decoder model and based on the content and the encoding of the plurality of action steps.

[0039] Clause 12: The method of clause 11 , further comprising, in response to receiving the reference data: pre-processing, with at least one processor, the reference data to conform to a predetermined data format, to produce formatted reference data; and dispatching, with at least one processor, the formatted reference data to at least one queue.

[0040] Clause 13: The method of clause 11 or clause 12, further comprising, in response to the formatted reference data being dispatched to the at least one queue: retrieving, with at least one processor and in queue order, the formatted reference data from the at least one queue; generating, with at least one processor, a plurality of tokens based on the formatted reference data; generating, with at least one processor, a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups; and converting, with at least one processor, the plurality of token groups into a plurality of representations using a distributed embedding model.

[0041] Clause 14: The method of any of clauses 1 1 -13, wherein the distributed embedding model is executed at least partly on a server cluster comprising at least one primary node and a plurality of secondary nodes, and wherein converting the plurality of token groups into the plurality of representations using the distributed embedding model further comprises, for each token group of the plurality of token groups: transmitting the token group to the primary node; determining a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group; directing the token group to the corresponding secondary node from the primary node; and generating a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

[0042] Clause 15: The method of any of clauses 1 1 -14, wherein each secondary node of the plurality of secondary nodes comprises a complete set of layers of the distributed embedding model, and wherein determining the corresponding secondary node further comprises determining the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

[0043] Clause 16: A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: receivereference data; store a representation of the reference data in a datastore; receive a prompt; determine a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action; retrieve a subset of the representation of the reference data from the datastore based on the first portion of the prompt; determine a generative output from a machine learning model based on an input comprising (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt; generate content responsive to the first portion of the prompt based on the generative output; determine an encoding of a plurality of action steps based on the second portion of the prompt; and execute the at least one executable action using a sequence-to-sequence decoder model and based on the content and the encoding of the plurality of action steps.

[0044] Clause 17: The computer program product of clause 16, wherein the program instructions further cause the at least one processor to, in response to receiving the reference data: pre-process the reference data to conform to a predetermined data format, to produce formatted reference data; and dispatch the formatted reference data to at least one queue.

[0045] Clause 18: The computer program product of clause 16 or clause 17, wherein the program instructions further cause the at least one processor to, in response to the formatted reference data being dispatched to the at least one queue: retrieve, in queue order, the formatted reference data from the at least one queue; generate a plurality of tokens based on the formatted reference data; generate a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups; and convert the plurality of token groups into a plurality of representations using a distributed embedding model.

[0046] Clause 19: The computer program product of any of clauses 16-18, wherein the distributed embedding model is executed at least partly on a server cluster comprising at least one primary node and a plurality of secondary nodes, and wherein the program instructions that cause the at least one processor to convert the plurality of token groups into the plurality of representations using the distributed embedding model further cause the at least one processor to, for each token group of the plurality of token groups: transmit the token group to the primary node; determine a corresponding secondary node of the plurality of secondary nodes based at leastpartly on a data modality of the token group; direct the token group to the corresponding secondary node from the primary node; and generate a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

[0047] Clause 20: The computer program product of any of clauses 16-19, wherein each secondary node of the plurality of secondary nodes comprises a complete set of layers of the distributed embedding model, and wherein the program instructions that cause the at least one processor to determine the corresponding secondary node further cause the at least one processor to determine the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

[0048] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Additional advantages and details are explained in greater detail below with reference to the non-limiting, exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0050] FIG. 1 is a schematic diagram of a system for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects;

[0051] FIG. 2 is a schematic diagram of example components of one or more devices of FIG. 1 , according to some non-limiting embodiments or aspects;

[0052] FIG. 3 is a flow diagram of a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects;

[0053] FIG. 4 is a flow diagram of a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects;

[0054] FIG. 5 is a flow diagram of a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects;

[0055] FIG. 6 is a schematic diagram of a system and method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects;

[0056] FIG. 7 is a schematic diagram of a system and method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects;

[0057] FIG. 8 is a schematic diagram of a system and method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects;

[0058] FIG. 9 is a schematic diagram of a system and method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects;

[0059] FIG. 10 is a schematic diagram of a system and method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects; and

[0060] FIG. 1 1 is a schematic diagram of a system and method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects.DETAILED DESCRIPTION

[0061] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to the embodiments as they are oriented in the drawing figures. However, it is to be understood that the present disclosure may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary and non-limiting embodiments or aspects of the disclosed subjectmatter. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0062] Some non-limiting embodiments or aspects are described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0063] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, and / or the like) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. In addition, reference to an action being “based on” a condition may refer to the action being “in response to” the condition. For example, the phrases “based on” and “in response to” may, in some non-limiting embodiments or aspects, refer to a condition for automatically triggering an action (e.g., a specific operation of an electronic device, such as a computing device, a processor, and / or the like).

[0064] As used herein, the term “communication” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of data (e.g., information, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / orrouted between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) that includes data. It will be appreciated that numerous other arrangements are possible.

[0065] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0066] As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the Internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.”

[0067] As used herein, the term “system” may refer to one or more computing devices or combinations of computing devices (e.g., processors, servers, client devices, software applications, components of such, and / or the like). Reference to “a device,” “a server,” “a processor,” and / or the like, as used herein, may refer to a previously recited device, server, or processor that is recited as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and the claims, a first device, a first server, or a first processor that is recited as performinga first step or a first function may refer to the same or different device, server, or processor recited as performing a second step or a second function.

[0068] The described systems, methods, and computer program products provide numerous technical improvements that address specific deficiencies in the described technical environments. For example, by receiving reference data, which includes batch data (e.g., data that is collected and released in batches, or groups, such as historic data) and stream data (e.g., data that is received in real-time with its generation / collection), in a multi-threaded communication channel and storing the reference data in a datastore for RAG model output, model latency may be reduced. Further to that end, batch data and stream data may be processed concurrently through multi-threading, which reduces data processing time. Moreover, the recency of reference data may be improved by ingesting stream data alongside batch data, which allows for generative output to accurately reflect real-time data. As used herein, “real-time” may refer to performance of a task or tasks during another process or before another process is completed, e.g., instantly or within milliseconds. In the context of processed data, real-time data processing may be timely and substantially synchronous with events that trigger the creation of the data. In the context of machine learning, real-time may denote the ability to perform data processing, analysis, and decision-making instantly or within milliseconds, ensuring that the output or action is delivered virtually immediately after the input is received, thus enabling timely and responsive operation.

[0069] The described systems and methods further provide the technical improvement of system scalability, e.g., improved performance with the increase of network demands and / or size of datastores, as reflected by computer resource metrics, such as reduced processing time, and as reflected by model performance metrics, such as increased accuracy and completeness of output. For example, by executing a distributed embedding model at least partly on a server cluster that includes at least one primary node and a plurality of secondary nodes, layers or sublayers of the distributed embedding model may be replicated and stored across the server cluster, which may improve performance through load-balancing. Additionally, the distributed embedding model may be configured to distribute inputs to secondary nodes based at least partly on the modality of a data chunk, which may allow for the processing of multi-modality prompts and generative output.

[0070] The described systems and methods further provide technically improved datastores for use in RAG models. For example, the use of a distributed hybrid feature datastore allows for the efficient storage of varied data and the fusion of various primary and secondary indexing techniques. These aspects, combined with parentchild retrieval processes, may lead to more accurate context for generative output, which may reduce model hallucination.

[0071] The described systems and methods further provide the technical benefit of combined generative output with executable actions. For example, the described systems and methods may determine portions of prompts that are associated with a generative prompt, and portions of prompts that are associated with executable actions. Portions associated with generative prompts may be routed to a RAG model process, to produce generative output. Portions associated with executable actions may be routed to a process that uses a sequence-to-sequence encoder and decoder model architecture, which may decode the encoding of a plurality of action steps that are based on the executable actions. Content based on generative output may then be combined with the executed actions to cause updates in one or more computing devices, connected computer systems, and / or networks, including in the user interface of a computing device of a user. As such, prompts may be processed in a unified and integrated manner, allowing for prompts that both call for generated content and seek the performance of actions in the computer environment.

[0072] In some non-limiting embodiments or aspects, the described systems and methods provide a configuration for continuous, multi-threaded contextual data sources, allowing for the asynchronous extraction and processing of data from multiple data sources. Furthermore, the described systems and methods are configured to tokenize, window, and dispatch reference data to one or more queues, from which a distributed embedding model may process the reference data in layers for indexing and data storage. These configurations allow for parallel processing and faster reactivity to user prompts. Indexed reference data may be stored in a distributed hybrid datastore, which promotes faster data retrieval and increased likelihood of models making connections between disparate reference data.

[0073] In some non-limiting embodiments or aspects, a separate hybrid datastore may maintain an action retriever, which may be implemented by extracting features from various applications and converting them into a latent space using graph attention transformers. This allows for a user interface to be digested into component elementse.g., user interactable and / or functional data elements, such as text fields, date pickers, checkboxes, buttons, sliders, frames, etc.) and relationships between those elements (e.g., rules and / or parameters defining how two or more interface elements interact with respect to each other and / or input data). User prompts may then be classified into generative and / or actionable prompts and the model output flow may be customized accordingly. For generative components of user prompts, the generative prompt may be searched in the distributed datastore and passed to a generative machine learning model. For actionable components of user prompts, the actions thereof may be performed and a response may be sent to the user. User prompts that include both generative and actionable components may be processed in a combined dataflow that includes a hybrid search of queries in the datastore and further includes a sequence-to-sequence model to integrate the generative output with executable steps within a user interface, after which a response may be sent to a user.

[0074] Referring now to FIG. 1 , shown is a schematic diagram of a system 100, according to some non-limiting embodiments or aspects. For example, devices, systems, and / or methods, as described herein, may be implemented in system 100. As shown in FIG. 1 , system 100 may include modeling system 102, datastore 104, computing device 106, and communication network 108. Modeling system 102, datastore 104, and computing device 106 may interconnect (e.g., establish a connection to communicate) via wired connections, wireless connections, or a combination of wired and wireless connections.

[0075] Modeling system 102 may include one or more computing devices configured to communicate with datastore 104 and / or computing device 106 at least partly over communication network 108. Modeling system 102 may be configured to receive data to train one or more machine learning models, train one or more machine learning models with received data, and use one or more trained machine learning models to generate an output. Modeling system 102 may include or be in communication with datastore 104.

[0076] Datastore 104 may include one or more computing devices configured to communicate with modeling system 102 and / or computing device 106 at least partly over communication network 108. Datastore 104 may be configured to store data used to train machine learning models, data used as input for trained machine learning models, and data including the output of trained machine learning models, in one or more non-transitory computer readable storage media. Datastore 104 maycommunicate with and / or be included in modeling system 102. Datastore 104 may include one or more datastores, such as a datastore for storing representations of token groups of reference data, a datastore for storing graph representations of user interfaces, a datastore for storing encodings of action steps for executable actions, a datastore for storing layers of a distributed embedding model, and / or the like.

[0077] Computing device 106 may include one or more processors that are configured to communicate with modeling system 102 and / or datastore 104 at least partly over communication network 108. Computing device 106 may be associated with a user and may include at least one user interface for transmitting data to and receiving data from modeling system 102 and / or datastore 104. For example, computing device 106 may show, on a display of computing device 106, one or more outputs of trained machine learning models executed by modeling system 102. By way of further example, one or more inputs for trained machine learning models may be determined or received by modeling system 102 based on input received from a user interface of computing device 106. In some non-limiting embodiments or aspects, computing device 106 may operate at least one user interface (e.g., including a web page, a software application, etc.), which may be used to transmit prompts to modeling system 102, may be acted upon by modeling system 102, may receive responses from modeling system 102, and / or the like.

[0078] Communication network 108 may include one or more wired and / or wireless networks over which the systems and devices of system 100 may communicate. For example, communication network 108 may include a cellular network (e.g., a longterm evolution (LTE®) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0079] The number and arrangement of systems and devices shown in FIG. 1 are provided as an example. There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or differently arranged systems and / or devices than those shown in FIG. 1. Furthermore, two or more systems or devices shown in FIG. 1 may be implemented within a single system ordevice, or a single system or device shown in FIG. 1 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another set of systems or another set of devices of system 100.

[0080] In some non-limiting embodiments or aspects, modeling system 102 may perform one or more methods for integrated processing of generative and instructive prompts in machine learning models. As used herein, a “prompt” (e.g., query, command, etc.) may include user input that is configured to cause content to be generated and / or at least one action to be executed. Each general prompt may include one or more specific prompts therein. Also as used herein, a portion of a prompt that is “generative” or “associated with a generative prompt” may refer to a portion of a prompt that calls for content to be generated based on the output of a generative machine learning model. Also as used herein, a portion of a prompt that is “instructive” or “associated with at least one executable action” may refer to a portion of a prompt that seeks the execution of one or more action steps. Actions may include, but are not limited to, the updating of the contents of a user interface, the manipulation of a user interface, the transmission of data by a computing device, the receipt of data by a computing device, the processing of data by a computing device, the display of data by a computing device, and / or the like. An action may be executed by performing one or more action steps. For example, the action of “sending an email” may be performed by a plurality of action steps, such as creating an email object, populating the recipient address field, populating the subject field, populating the body field, and triggering the sending of the email object. By way of another example, the action of “set my status to ‘away’” may be performed by a single action step, such as pushing an update message to a server that is configured to update the status field of a user record to “away.”

[0081] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to receive reference data (e.g., contextual data, a RAG model corpus, etc.). The reference data may include batch data and / or stream data. For example, at least one processor of modeling system 102 may be configured to receive batch data and stream data in separate threads of a multithreaded communication channel (e.g., communicative connection to one or more data sources that is configured for parallel processing of batch data and stream data).

[0082] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to, in response to receiving the reference data, pre-process and dispatch the reference data. For example, the at least one processor may be configured to pre-process the reference data to conform to a predetermined data format to produce formatted reference data. Such pre-processing may include, but is not limited to, data cleaning (e.g., imputing missing values, removing duplicates, detecting and removing outliers, etc.), data transformation (e.g., normalization, scaling, transformations to correct for variance or skew, etc.), feature engineering (e.g., creating features based on the reference data), reformatting (e.g., translating variables from one variable type to another), text processing, dimensional reduction, and / or the like. The result of pre-processing may include the production of formatted reference data. The at least one processor may be further configured to dispatch the formatted reference data to at least one queue (e.g., an ordered data pipeline) for downstream tasks, including tokenization, grouping, and embedding.

[0083] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to, in response to the formatted reference data being dispatched to the at least one queue, perform a series of steps. For example, the at least one processor may be configured to retrieve, in queue order (e.g., first-in-first-out), the formatted reference data from the at least one queue. The at least one processor may be further configured to generate a plurality of tokens based on the formatted reference data. Tokenization may include converting the reference data into a structured format for easier analysis, storage, and / or processing. Tokens of reference data may include units of data, e.g., words, sub-words, characters, meaningful elements, and / or the like, representative of the context and / or meaning of the tokenized reference data. The at least one processor may further generate a plurality of token groups based on the plurality of tokens. A token group may include two or more tokens. In some non-limiting embodiments or aspects, each token group of the plurality of token groups may be non-overlapping, e.g., sharing no tokens in common with any other token group. Alternatively, each token group of the plurality of token groups may share at least one token with another token group of the plurality of token groups (e.g., applying an overlapping window technique). The size of the overlap (e.g., number of tokens in common) between token groups may be determined by a token limit (e.g., a predetermined hyperparameter) of a distributed embedding model. The at least one processor may further convert the plurality oftoken groups into a plurality of representations e.g., embeddings, vector representations, latent representations, etc.) using the distributed embedding model.

[0084] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to execute the distributed embedding model at least partly on a server cluster. The server cluster may include at least one primary node and a plurality of secondary nodes. When converting the plurality of token groups into the plurality of representations using the distributed embedding model, the at least one processor may be further configured to, for each token group of the plurality of token groups: transmit the token group to at least one primary node; determine (e.g., using or in concert with the primary node) a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group, wherein the corresponding secondary node is configured to process data having the data modality of the given token group; direct (e.g., using or in concert with the primary node) the token group to the corresponding secondary node from the primary node; and generate a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

[0085] In some non-limiting embodiments or aspects, each secondary node of the plurality of secondary nodes may include a complete set of layers of the distributed embedding model. In such a scenario, when determining the corresponding secondary node, at least one processor of modeling system 102 may be configured to determine the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster (e.g., sending the input data for the distributed embedding model to the secondary node with the least load, as determined by the node in the server cluster having least memory in active use). Alternatively, each secondary node of the plurality of secondary nodes may include a set of layers of the distributed embedding model, but not a complete set of layers. In such a scenario, when determining the corresponding secondary node, the at least one processor may be configured to determine the corresponding secondary node of the plurality of secondary nodes based at least partly on load balance of the server cluster and the required set of layers. Alternatively, some of the secondary nodes may include a complete set of layers, and some of the secondary nodes may include a partial set of layers. In such a scenario, when determining the corresponding secondary node, the at least one processor may be configured to determine thecorresponding secondary node of the plurality of secondary nodes based at least partly on load balance of the server cluster, and if the least loaded secondary node has a partial set of layers, further based at least partly on the required set of layers. See FIG. 9 for further detailed description related to implementing a distributed embedding model.

[0086] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to store a representation (e.g., embedding, vector representation, latent representation) of the reference data in a datastore. For example, the representation of the reference data may include the plurality of representations converted from a plurality of groups of tokens that are generated based on the reference data. Storing the representation of the reference data in a datastore may include storing the plurality of representations in the datastore. In some non-limiting embodiments or aspects, the datastore may include a distributed datastore having primary datastore nodes and secondary datastore nodes, for use with primary-secondary indexing techniques. See FIG. 7 for further detailed description related to storing reference data (e.g., as representations of tokens) in a distributed datastore using primary-secondary indexing, and FIG. 11 for further detailed description related to retrieving representations of token groups of reference data from a distributed datastore using primary-secondary indexing.

[0087] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to receive a prompt. For example, the at least one processor may be configured to receive a prompt from computing device 106 associated with a user. The prompt may be received via a user interface (e.g., a webpage, a software application, etc.) operated on computing device 106. The prompt may include at least one portion that is associated with one or more generative prompts. Additionally or alternatively, the prompt may include at least one portion that is associated with at least one executable action.

[0088] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to extract a plurality of features based on the user interface operated on computing device 106. For example, information about the user interface may be transmitted to modeling system 102, e.g., through an edge computing process in concert with the at least one processor. By way of further example, the at least one processor may be configured to extract a plurality of features based on the user interface using a neural network model. The plurality of featuresmay be associated with (i) a plurality of elements of the user interface (e.g., including one or more buttons, fields, menus, checkboxes, sliders, date pickers, interactive form elements, and / or the like) and (ii) a plurality of relationships between the plurality of elements (e.g., dependencies, operative order, triggered updates between elements, associations, etc.). In some non-limiting embodiments or aspects, the at least one processor may be configured to construct a graph (e.g., a graphical representation) of the user interface, wherein the graph includes a plurality of nodes and a plurality of edges. Each node of the plurality of nodes may be associated with an element of the plurality of elements of the user interface. Each edge of the plurality of edges may be associated with a relationship (of the plurality of relationships) between two nodes of the plurality of nodes that are connected by the edge.

[0089] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to determine a first portion of the prompt that is associated with at least one generative prompt. Additionally or alternatively, the at least one processor may be configured to determine a second portion of the prompt that is associated with at least one executable action. (See FIG. 8 for further detailed description related to ingesting a prompt and processing the prompt according to generative or instructive portions). To illustrate, the user prompt may include the natural language input: “Send me an email with a brief summary of last week’s meetings.” A first portion of the prompt may be associated with a generative prompt, e.g., a request to generate a brief summary of meetings for the user that occurred in the last week. A second portion of the prompt may be associated with at least one executable action, e.g., sending the user an email that includes the brief summary.

[0090] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to retrieve a subset of the representation of the reference data from the datastore based on the first portion of the prompt (e.g., associated with the at least one generative prompt). The at least one processor may be configured to retrieve the subset of the representation of the reference data by retrieving, from a distributed datastore, the representations of relevant token groups of the tokenized reference data, based on a primary-secondary indexing hybrid search of the distributed datastore. See FIG. 10 for further detailed description related to performing a hybrid search of primary-secondary indexed representations of token groups.

[0091] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to determine a generative output from a machine learning model (e.g., a RAG model) based on an input including the subset of the representation of the reference data. The input may also include at least part of the first portion of the prompt that is associated with at least one generative prompt. To return to the prior illustration, the retrieved subset of the representation of the reference data may be related to reference data associated with the user’s meetings that occurred in the last week. Such reference data may be sourced from calendar data objects, saved notes files, post-meeting emails, and / or the like. This reference data may act as the context for the RAG model. The portion of the prompt associated with a generative prompt, “a brief summary of last week’s meetings”, may also be provided as input to the RAG model, to direct the generative model to generate a brief summary that is based on the retrieved subset of the representation of the reference data. The at least one processor may then generate content responsive to the first portion of the prompt based on the generative output of the machine learning model. In some non-limiting embodiments or aspects, the generated content may include the generative output. Additionally or alternatively, the generated content may include a portion of the generative output, reformatted generative output, stylized generative output, and / or the like.

[0092] In some non-limiting embodiments or aspects, at least one processor of modeling system 102 may be configured to determine an encoding (e.g., vector representation) of a plurality of action steps based on the second portion of the prompt, which is associated with at least one executable action. For example, the at least one processor may be configured to convert the second portion of the prompt into a vector representation of the plurality of action steps using a transformer encoder model (e.g., a neural network model configured to handle sequential data by using self-attention mechanisms to capture dependencies between elements in the sequence). The at least one processor may also be configured to determine the encoding of the plurality of action steps using a sequence-to-sequence encoder model (e.g., a seq2seq encoder, a neural network architecture configured to transform one sequence of data into another encoded sequence) and based on the vector representation (e.g., as input to the sequence-to-sequence encoder model). The vector representation may include one or more representations associated with the plurality of action steps.

[0093] In some non-limiting embodiments or aspects, the at least one processor of modeling system 102 may be configured to execute the at least one executable action. For example, the at least one processor may be configured to execute the at least one executable action using a sequence-to-sequence decoder model (e.g., a seq2seq decoder, a neural network architecture configured to transform one sequence of data into another decoded sequence) and based on the encoding of the plurality of action steps (e.g., as input to the sequence-to-sequence decoder model) and the generated content. It will be appreciated that a sequence-to-sequence encoder model and a sequence-to-sequence decoder model may be included together in a same sequence- to-sequence neural network architecture. In some non-limiting embodiments or aspects, when executing the at least one executable action, the at least one processor may be further configured to traverse the graph of the user interface using the sequence-to-sequence decoder model and based on the encoding of the plurality of action steps. See FIG. 1 1 for further detailed description related to generating a graph representation of a user interface and executing one or more actions by traversing the graph representation.

[0094] Referring now to FIG. 2, shown is a diagram of example components of a device 200, according to non-limiting embodiments. Device 200 may correspond to modeling system 102, datastore 104, or computing device 106, as an example. In some non-limiting embodiments, such systems or devices may include at least one device 200 and / or at least one component of device 200. The number and arrangement of components shown are provided as an example. In some non-limiting embodiments, device 200 may include additional components, fewer components, different components, or differently arranged components than those shown. Additionally, or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0095] As shown in FIG. 2, device 200 may include a bus 202, a processor 204, memory 206, a storage component 208, an input component 210, an output component 212, and a communication interface 214. Bus 202 may include a component that permits communication among the components of device 200. In some non-limiting embodiments, processor 204 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processingunit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 206 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 204.

[0096] With continued reference to FIG. 2, storage component 208 may store information and / or software related to the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid-state disk, etc.) and / or another type of computer-readable medium. Input component 210 may include a component that permits device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally, or alternatively, input component 210 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 212 may include a component that provides output information from device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 214 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 214 may permit device 200 to receive information from another device and / or provide information to another device. For example, communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0097] Device 200 may perform one or more processes described herein. Device 200 may perform these processes based on processor 204 executing software instructions stored by a computer-readable medium, such as memory 206 and / or storage component 208. A computer-readable medium may include any non- transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physicalstorage devices. Software instructions may be read into memory 206 and / or storage component 208 from another computer-readable medium or from another device via communication interface 214. When executed, software instructions stored in memory 206 and / or storage component 208 may cause processor 204 to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software. The term “configured to,” as used herein, may refer to an arrangement of software, device(s), and / or hardware for performing and / or enabling one or more functions (e.g., actions, processes, steps of a process, and / or the like). For example, “a processor configured to” may refer to a processor that executes software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0098] Referring now to FIG. 3, shown is a flow diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 300 shown in FIG. 3 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102.

[0099] As shown in FIG. 3, at step 302, process 300 may include receiving reference data. For example, modeling system 102 may receive reference data. In some non-limiting embodiments or aspects, the reference data may include batch data and stream data. When receiving the reference data, modeling system 102 may receive the batch data and the stream data in separate threads of a multi-threaded communication channel.

[0100] As shown in FIG. 3, at step 304, process 300 may include storing a representation of the reference data. For example, modeling system 102 may store arepresentation of the reference data in a datastore. In some non-limiting embodiments or aspects, the representation of the reference data stored at step 304 may include the plurality of representations produced at step 412 of process 400 (see FIG. 4). For example, when storing the representation of the reference data, modeling system 102 may store the plurality of representations in the datastore.

[0101] As shown in FIG. 3, at step 306, process 300 may include receiving a prompt. For example, modeling system 102 may receive a prompt. In some nonlimiting embodiments or aspects, the prompt may include a portion associated with a generative prompt. Additionally or alternatively, the prompt may include a portion associated with at least one executable action.

[0102] As shown in FIG. 3, at step 308, process 300 may include determining a first portion and a second portion of the prompt. For example, modeling system 102 may determine a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action.

[0103] As shown in FIG. 3, at step 310, process 300 may include retrieving a subset of the representation of the reference data. For example, modeling system 102 may retrieve a subset of the representation of the reference data from the datastore based on the first portion of the prompt. In some non-limiting embodiments or aspects, the subset of the representation of the reference data may include one or more representations of one or more token groups of tokenized reference data.

[0104] As shown in FIG. 3, at step 312, process 300 may include determining a generative output. For example, modeling system 102 may determine a generative output from a machine learning model based on an input including (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt.

[0105] As shown in FIG. 3, at step 314, process 300 may include generating content based on the generative output. For example, modeling system 102 may generate content responsive to the first portion of the prompt based on the generative output. In some non-limiting embodiments or aspects, the generated content based on the generative output may include or be derived from the generative output.

[0106] As shown in FIG. 3, at step 316, process 300 may include determining an encoding of the plurality of action steps. For example, modeling system 102 may determine an encoding of a plurality of action steps based on the second portion ofthe prompt. In some non-limiting embodiments or aspects, determining the encoding of the plurality of action steps based on the second portion of the prompt may include, when determining the encoding of the plurality of action steps, converting the second portion of the prompt into a vector representation using a transformer encoder model. Modeling system 102 may further determine the encoding of the plurality of action steps using a sequence-to-sequence encoder model and based on the vector representation.

[0107] As shown in FIG. 3, at step 318, process 300 may include executing one or more executable actions based on the content and the encoding of the plurality of action steps. For example, modeling system 102 may execute the at least one executable action using a sequence-to-sequence decoder model and based on the content and the encoding of the plurality of action steps.

[0108] Referring now to FIG. 4, shown is a flow diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 400 shown in FIG. 4 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 400 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 400 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102. The steps of process 400 may augment the steps of process 300 depicted in FIG. 3.

[0109] As shown in FIG. 4, at step 302, process 400 may include receiving reference data. For example, modeling system 102 may receive reference data, which may include batch data and stream data.

[0110] As shown in FIG. 4, at step 402, process 400 may include pre-processing the reference data. For example, modeling system 102 may, in response to receiving the reference data at step 302, pre-process the reference data to conform to a predetermined data format, to produce formatted reference data.

[0111] As shown in FIG. 4, at step 404, process 400 may include dispatching the formatted reference data to a queue. For example, modeling system 102 may, in response to receiving the reference data at step 302, and subsequent to preprocessing the reference data at step 404, dispatch the formatted reference data to at least one queue. In some non-limiting embodiments or aspects, segments of the formatted reference data may be processed in the at least one queue in queue order (e.g., first-in-first-out).

[0112] As shown in FIG. 4, at step 406, process 400 may include retrieving the formatted reference data from the queue. For example, modeling system 102 may, in response to the formatted reference data being dispatched to the at least one queue at step 404, retrieve, in queue order, the formatted reference data from the at least one queue.

[0113] As shown in FIG. 4, at step 408, process 400 may include generating a plurality of tokens based on the formatted reference data. For example, modeling system 102 may, in response to the formatted reference data being dispatched to the at least one queue at step 404, and subsequent to retrieving the formatted reference data at step 406, generate a plurality of tokens based on the formatted reference data.

[0114] As shown in FIG. 4, at step 410, process 400 may include generating a plurality of token groups based on the plurality of tokens. For example, modeling system 102 may, in response to the formatted reference data being dispatched to the at least one queue at step 404, and subsequent to generating the plurality of tokens at step 408, generate a plurality of token groups based on the plurality of tokens. In some non-limiting embodiments or aspects, each token group of the plurality of token groups may share at least one token with another token group of the plurality of token groups (e.g., applying an overlapping window technique).

[0115] As shown in FIG. 4, at step 412, process 400 may include converting the plurality of token groups into a plurality of representations. For example, modeling system 102 may, in response to the formatted reference data being dispatched to the at least one queue at step 404, and subsequent to generating a plurality of token groups at step 410, convert the plurality of token groups into a plurality of representations using a distributed embedding model.

[0116] In some non-limiting embodiments or aspects, the representation of the reference data that is stored at step 304 of process 300 (see FIG. 3) may include the plurality of representations produced at step 412 of process 400. For example, whenstoring the representation of the reference data at step 304 of process 300, modeling system 102 may store the plurality of representations in the datastore.

[0117] In some non-limiting embodiments or aspects, the distributed embedding model may be executed at least partly on a server cluster including at least one primary node and a plurality of secondary nodes. When converting the plurality of token groups into the plurality of representations using the distributed embedding model, at step 412, modeling system 102 may be configured to perform a series of steps for each token group of the plurality of token groups. For example, modeling system 102 may transmit the token group to the at least one primary node, determine a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group, direct the token group to the corresponding secondary node from the primary node, and generate a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster. The at least one primary node may be included in performing the above data management steps vis-a-vis the secondary nodes.

[0118] In some non-limiting embodiments or aspects, each secondary node of the plurality of secondary nodes may include a complete set of layers of the distributed embedding model. For example, when determining the corresponding secondary node, modeling system 102 may determine the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

[0119] In some non-limiting embodiments or aspects, each secondary node of the plurality of secondary nodes may include a (e.g., partial) set of layers of the distributed embedding model. A set of secondary nodes of the plurality of secondary nodes may, in combination, represent a complete set of layers of the distributed embedding model. For example, a first secondary node may include a first set of layers of the distributed embedding model and may receive an initial input from the primary node. The output of the first secondary node may be the input of a second secondary node that includes a second set of layers of the distributed embedding model. The output of the second secondary node may be the input of a third secondary node that includes a third set of layers of the distributed embedding model. This sequence may be repeated for n secondary nodes, until the ultimate secondary node, which provides its output (e.g., the output of the final layer of the distributed embedding model) back to the primarynode. In this manner, two or more secondary nodes, each with a set of layers of the distributed embedding model, may be combined to represent a complete set of layers of the distributed embedding model.

[0120] Referring now to FIG. 5, shown is a flow diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 500 shown in FIG. 5 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 500 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 500 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102. The steps of process 400 may augment the steps of process 300 depicted in FIG. 3.

[0121] As shown in FIG. 5, at step 306, process 500 may include receiving a prompt. For example, modeling system 102 may receive a prompt. In some nonlimiting embodiments or aspects, modeling system 102 may receive the prompt via a user interface of computing device 106 of a user.

[0122] As shown in FIG. 5, at step 502, process 500 may include extracting a plurality of features based on the user interface. For example, modeling system 102 may extract a plurality of features based on the user interface using a neural network model. In some non-limiting embodiments or aspects, the plurality of features may be associated with (i) a plurality of elements of the user interface and (ii) a plurality of relationships between the plurality of elements.

[0123] As shown in FIG. 5, at step 504, process 500 may include constructing a graph of the user interface based on the plurality of features. For example, modeling system 102 may construct a graph of the user interface based on the plurality of features. In some non-limiting embodiments or aspects, each node of the plurality of nodes may be associated with an element of the plurality of elements. Additionally or alternatively, each edge of the plurality of edges may be associated with a relationship of the plurality of relationships.

[0124] As shown in FIG. 5, at step 318, process 500 may include executing one or more executable actions based on the content and the encoding of the plurality of actions steps. For example, modeling system 102 may execute the at least one executable action based on the content and the encoding of the plurality of action steps.

[0125] As shown in FIG. 5, at step 506, process 500 may include traversing the graph using a sequence-to-sequence decoder model. For example, modeling system 102 may, when executing the at least one executable action at step 318, traverse the graph of the user interface at step 506 using the sequence-to-sequence decoder model and based on the encoding of the plurality of action steps.

[0126] Referring now to FIG. 6, shown is a schematic diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 600 shown in FIG. 6 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 600 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 600 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102.

[0127] As shown in FIG. 6, process 600 may include use of a multi-threaded communication channel 601 . Multi-threaded communication channel 601 may include at least one thread for processing batch data 602 and at least one thread for processing stream data 608. For example, modeling system 102 may perform a continuous, multi-threaded intake process to asynchronously extract data from a variety of sources, include batch data 602 sources and stream data 608 sources. Each thread of multi-threaded communication channel 601 may ingest data from a data source, pre-process the data, and dispatch the data to at least one queue. For example, modeling system 102 may ingest batch data 602 in at least one thread of multi-threaded communication channel 601 , conform batch data 602 to a predetermined format using pre-processing filter 604, and send the pre-processedbatch data 602 to a first queue 606 for downstream processing. By way of further example, modeling system 102 may ingest stream data 608 in at least one thread of multi-threaded communication channel 601 , conform stream data 608 to a predetermined format using pre-processing filter 610, and send the pre-processed stream data 608 to a second queue 612 for downstream processing.

[0128] As shown in FIG. 6, process 600 may further include a tokenizing process 614. For example, modeling system 102 may use a tokenizer function to interface with each queue of the plurality of queues of pre-processed data, including first queue 606 and second queue 612. Modeling system 102 may retrieve lines of data from first queue 606 and second queue 612, and further break down and convert the lines of data into individual tokens.

[0129] As shown in FIG. 6, process 600 may further include a token grouping process 616. For example, modeling system 102 may group (e.g., chunk) tokens using an overlapping window technique, such that each token group shares at least one token with another token group. The size / overlap of the window between token groups may be determined, at least partly, by the token limit of the embedding model. Modeling system 102 may further publish the token groups to queue 618. In some non-limiting embodiments or aspects, token grouping process 616 may be performed as part of tokenizing process 614.

[0130] As shown in FIG. 6, process 600 may further include operating a distributed embedding model 620. For example, modeling system 102 may retrieve token groups from queue 618 and store the token groups in a distributed datastore by using distributed embedding model 620. In some non-limiting embodiments or aspects, modeling system 102 may, using distributed embedding model 620, replicate and store the token chunks across a plurality of computing devices in a distributed datastore. See FIGS. 7 and 9 for further detailed description of implementing distributed embedding model 620.

[0131] Referring now to FIG. 7, shown is a schematic diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 700 shown in FIG. 7 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completionof a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 700 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 700 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102. Process 700, shown in FIG. 7, may augment process 600, shown in FIG. 6.

[0132] As shown in FIG. 7, process 700 may include operating a distributed embedding model 620. For example, modeling system 102 may retrieve token groups from queue 618 and store the token groups in a distributed datastore by using distributed embedding model 620. In some non-limiting embodiments or aspects, modeling system 102 may, using distributed embedding model 620, replicate and store the token chunks across a plurality of computing devices in a distributed datastore. Each layer of distributed embedding model 620 may compute the operations related to its assigned portion of the model and pass the output as an input to the next layer, finally sending back the generated output via at least one primary node for indexing, in indexing process 702. The primary node of the server cluster operating distributed embedding model 620 may orchestrate multimodality, e.g., identifying data chunk type (e.g., image, video, audio, text, etc.) and directing the data chunk to the appropriate encoder in the distributed data store. Indexing process 702 may store the index output in distributed datastore 703. Distributed datastore 703 may include at least one primary datastore node 704 and a plurality of secondary datastore nodes 706.

[0133] In some non-limiting embodiments or aspects, indexing process 702 may leverage a fusion of various primary and secondary indexing techniques with parentchild group retrieval. For example, a primary index may be used to link a stored data chunk to an associated primary datastore node 704, and a secondary index may be used to link the data chunk from the associated primary datastore node 704 to an associated secondary datastore node 706. Such a primary-secondary indexing allows for more rapid retrieval of reference data from distributed datastore 703. See FIG. 10 for further detailed description related to primary-secondary indexing for use with distributed datastore 703.

[0134] Additionally or alternatively, modeling system 102 may store data of user interface features (e.g., interface elements) and relationships of a user interface (e.g.,an application, a website, etc.) in a hybrid feature store, which may be the same or different datastore as distributed datastore 703. An action encoder may determine relationships between various features of the user interface to form a graph, and convert the graph into a latent space using graph attention transformers. See FIG. 11 for further detailed description related to feature and relationship extraction, graph generation, and action encoding and decoding.

[0135] Referring now to FIG. 8, shown is a schematic diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 800 shown in FIG. 8 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 800 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 800 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102.

[0136] As shown in FIG. 8, process 800 may include receiving a prompt from computing device 106 of user 801 . For example, modeling system 102 may receive a prompt via a user interface from computing device 106 of user 801 . In some nonlimiting embodiments or aspects, the prompt may be configured in a natural language text format. Additionally or alternatively, the prompt may include at least a portion of data in other modalities, such as image, video, audio, and / or the like.

[0137] As shown in FIG. 8, process 800 may include, at step 802, checking a cache for existence of a same or similar (e.g., semantically equivalent) prompt. For example, modeling system 102 may query a datastore including a cache including non-transitory memory that stores data of historic prompts received from one or more users, including user 801. If modeling system 102 determines that a same or similar prompt was previously received and processed, modeling system 102 may retrieve the prior generated response from the cache and provide the prior generate response in response 804 to user 801 . If modeling system 102 determines that a same or similarprompt was not previously received or processed, modeling system 102 may proceed to generate new response 804 by proceeding to a classification process in step 806.

[0138] As shown in FIG. 8, process 800 may include, at step 806, classifying the prompt into one or more generative and / or actionable components. For example, modeling system 102 may determine at least one first portion of the prompt associated with at least one generative prompt. Additionally or alternatively, modeling system 102 may determine at least one second portion of the prompt associated with at least one executable action. For each portion of the prompt associated with a generative prompt, modeling system 102 may initiate a generative process, at step 810. For each portion of the prompt associated with an executable action, modeling system 102 may initiate an actionable process, at step 808.

[0139] As shown in FIG. 8, process 800 may include, at step 814, processing each portion of the prompt associated with a generative prompt using a distributed embedding model 814. For example, modeling system 102 may use a distributed embedding model 814 to determine a representation of the generative portions, which may be used for querying and retrieving reference data from a distributed datastore.

[0140] As shown in FIG. 8, process 800 may include, at step 816, performing a hybrid search of a distributed datastore, based on the representation of the generative portions of the prompt. For example, modeling system 102 may perform a hybrid search of a distributed datastore that stores reference data (e.g., in the form of a representation of one or more token groups) for use in RAG models. The format of the retrieved reference data may include the representation of one or more token groups (e.g., chunks), such that modeling system 102 may retrieve a subset of the representation of the body of stored reference data. Modeling system 102 may provide the retrieved reference data, at step 818, to one or more generative machine learning models, to generate at least one output in step 820.

[0141] As shown in FIG. 8, process 800 may include, at step 820, determining a generative output from one or more machine learning models. For example, modeling system 102 may determine a generative output from one or more generative machine learning models based on an input that includes the subset of the representation of the reference data that was retrieved at step 818. Additionally or alternatively, the input may include one or more portions of the prompt, including one or more generative portions, one or more actionable portions, and / or the like. The output of the one or more generative machine learning models may be used to generate contentresponsive to the generative portions of the prompt. For example, the content may include the generative output, in whole or in part. By way of further example, the generative output may be further processed, formatted, and / or the like, to produce the generated content.

[0142] As shown in FIG. 8, process 800 may include, at step 822, assessing the generated content for any further action steps. For example, the generated content may itself include reference to executable actions, which may or may not have been included in the original user prompt. Any further executable actions that may be included in the generated content can be likewise processed like the actionable portions of the prompt, as initiated in step 808. For example, modeling system 102 may assess the generated content, identifying one or more action steps, and forward actionable portions of the generated content to an encoder in step 812.

[0143] As shown in FIG. 8, process 800 may include, at step 812, encoding one or more action steps, e.g., of at least one executable action. The action steps may be based on one or more portions of the prompt that are associated with at least one executable action. Additionally or alternatively, the action steps may be based on one or more action steps that were included in the generated content from step 820. In some non-limiting embodiments or aspects, modeling system 102 may use a transformer encoder model to translate natural language into vector representation to capture the intent and context of the portions of the prompt associated with at least one executable action. In this manner, the encoding of the plurality of action steps may include vector representations.

[0144] As shown in FIG. 8, process 800 may include, at step 824, translating (e.g., converting) vector representations of one or more steps of at least one executable action. For example, modeling system 102 may use a sequence-to-sequence decoder model to translate vector representations (e.g., the encoded portions of the prompt associated with at least one executable action), from step 812 into a sequence of steps for at least one executable action that may manipulate the user interface. Using the sequence-to-sequence decoder model and a previously generated graph representation of the user interface, the at least one executable action may be executed by decoding the encoding of a plurality of action steps to traverse the graph representation of the user interface and cause one or more actions to be executed. See FIG. 11 for further detailed description related to constructing and traversing a graph representation of a user interface. The at least one executable action may befurther executed based on the generated content, such as by inputting the content to the interface, manipulating the interface based on the content, and / or the like.

[0145] As shown in FIG. 8, process 800 may include, at step 828, generating an action response. For example, modeling system 102 may generate a response (e.g., including a message including text data, audio data, video data, image data, etc.) to computing device 106 of user 801 to indicate that the prompt has been acted on. For example, the action response may include a description of the one or more actions that have been executed. Additionally or alternatively, the action response may include the generated content. In some non-limiting embodiments or aspects, the user interface may include a fillable form. The form, including one or more text fields, date pickers, checkboxes, radio buttons, and / or the like, may be manipulated, selected, populated, and / or the like, based on prompt of user 801 (e.g., including the generated content).

[0146] Referring now to FIG. 9, shown is a schematic diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 900 shown in FIG. 9 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 900 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 900 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102.

[0147] As shown in FIG. 9, process 900 illustrates operation of a distributed embedding model on server cluster 910. Server cluster 910 may include at least one primary node 912 and a plurality of secondary nodes, such as secondary node 914, secondary node 916, and secondary node 918. The distributed embedding model may include a plurality of model layers 902, which are represented by layers L, X, K, N and M. Each layer of plurality of model layers 902 may accept an input (e.g., an original input to the model, input received from another layer, etc.), perform at least one operation on the input, and produce an output (e.g., a final output of the model,output to another layer, etc.). For example, distributed embedding model may include one or more of the following layers: embedding layers (e.g., to convert an input to a vector representation of the input), convolutional layers (e.g., to capture local patterns), recurrent layers (e.g., to capture dependencies), transformer layers (e.g., to capture dependencies, compute attention scores, etc.), fully connected layers (e.g., to combine features extracted from previous layers), normalization layers (e.g., to normalize input to stabilize learning), dropout layers (e.g., to prevent overfitting), pooling layers (e.g., to reduce dimensions), activation layers (e.g., to improve the computational efficiency of the model), or any combination thereof.

[0148] In some non-limiting embodiments or aspects, each secondary node 914, 916, 918 may include a complete set (e.g., a full copy) of the plurality of layers 902, such that the entirety of the distributed embedding model may be executed on any given secondary node 914, 916, 918. In this manner, primary node 912 may primarily act as a load balancer to direct data to the secondary node that is currently under the least load.

[0149] In some non-limiting embodiments or aspects, plurality of layers 902 may be divided up and assigned to a plurality of secondary nodes (e.g., secondary nodes 914, 916, 918). For example, plurality of model layers 902 may be split into two or more sets (e.g., groups) of layers, e.g., a first set of layers 904, a second set of layers 906, and a third set of layers 908. Each set of layers may include the entirety of each layer therein. Alternatively, one or more of the sets of layers may include a part of a layer (e.g., sub-layer) that corresponds to a complimentary part of a layer (e.g., sublayer) that is included in another set of layers. For example, first set of layers 904 (also designated by the letter “A”), may include the entirety of layer L and a part of layer X of the distributed embedding model. Second set of layers 906 (also designated by the letter “B”) may include the entirety of layer K and parts of layers X and N. Third set of layers 908 (also designated by the letter “C”) may include the entirety of layer M and a part of layer N. The part of layer X in set of layers 904 may be complimentary to the part of layer X in set of layers 906, such that each part, when combined, forms the whole of layer X. Likewise, the part of layer N in set of layers 906 may be complimentary to the part of layer N in set of layers 908, such that each part, when combined, forms the whole of layer N.

[0150] In some non-limiting embodiments or aspects, sets of layers 904, 906, 908 may be stored on one or more secondary nodes 914, 916, 918 of server cluster 910.Each secondary node may include one or more sets of layers. For example, first secondary node 914 may include a copy of first set of layers 904 (“A”) and a copy of third set of layers 908 (“C”), second secondary node 916 may include a copy of first set of layers 904 (“A”) and a copy of second set of layers 906 (“B”), and third secondary node 918 may include a copy of second set of layers 906 (“B”) and a copy of third set of layers 908 (“C”). The output from one set of layers and input to another (e.g., values, vectors, error gradients, etc.) may include the passing of data from one secondary node to another secondary node, which may be facilitated by primary node 912. For example, primary node 912 may route input to first set of layers 904 (“A”) that is operated on first secondary node 914; then, primary node 912 may route the output of first set of layers 904 (“A”) that is operated on first secondary node 914 to be input to second set of layers 906 (“B”) that is operated on second secondary node 916; then, primary node 912 may route the output of second set of layers 906 (“B”) that is operated on second secondary node 916 to be input to third set of layers 908 (“C”) that is operated on third secondary node 918; then, primary node 912 may receive the output of third set of layers 908 (“C”) that is operated on third secondary node 918 and return the output as a final output of the distributed embedding model. It will be appreciated that because primary node 912 may act as a load balancer, primary node 912 may direct inputs and outputs to secondary nodes with the least load. For example, and purely for illustrative purposes, viable routes through server cluster 910, to make use of all of the plurality of layers 902 by traversing sets of layers 904, 906, 908 (e.g., from set “A” to set “B” to set “C”), may include: 914 to 916 to 918; 914 to 918 to 918; 914 to 918 to 914; 914 to 916 to 914; 916 to 916 to 918; 916 to 918 to 918; 916 to 916 to 914; or 916 to 918 to 914. Such a configuration provides parallel processing and load balancing efficiencies and allows for the operation of large and complex models by distribution of a server cluster 910.

[0151] In some non-limiting embodiments or aspects, distributed embedding model may include different versions of sets of layers based on data modality. For example, distributed embedding model may include a first version of second set of layers 906, where layer K is configured to process textual data. Distributed embedding model may include a second version of second set of layers 906, where layer K is configured to process image data. Distributed embedding model may include a third version of second set of layers 906, where layer K is configured to process audio data. It will be appreciated that various versions of one or more sets of layers may be configured toprocess different modalities of data, and primary node 912 of server cluster 910 may route data to appropriate secondary nodes of server cluster 910 based on the modality of the data and the secondary nodes that store layers configured for that modality. Primary node 912 may further route data based on both modality and node load.

[0152] Referring now to FIG. 10, shown is a schematic diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 1000 shown in FIG. 10 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 1000 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 1000 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102.

[0153] As shown in FIG. 10, process 1000 may include a process for retrieving reference data from a datastore for use in RAG models. Process 1000 may include, at step 1002, receiving a prompt. For example, modeling system 102 may receive a prompt from computing device 106 of a user 801 . At least a portion of prompt may call for generated content. Modeling system 102 may retrieve related reference data to augment one or more generative machine learning models. The reference data may be previously tokenized, grouped (e.g., chunked) into token groups, converted to vector representations of each token group, and stored in a distributed datastore based on primary indexes 1004 and secondary indexes 1006. Primary indexes 1004 may store the first level of mapping to secondary indexes 1006 to enable modeling system 102 to swiftly locate the necessary representation of the token group of the reference data within the distributed datastore. Primary indexes 1004 may be stored on one or more computing devices of the datastore and may include summaries, key metrics, or granular chunks of the token groups, to promote efficient data retrieval. Secondary indexes 1006 may be stored on one or more computing devices of the datastore and may include complete representations of token groups.

[0154] In some non-limiting embodiments or aspects, modeling system 102 may further refine the above-described arrangement by grouping representations of token groups within the secondary index 1006. For example, two or more representations of token groups stored in secondary indexes 1006 may be collected into a group of representations of token groups, which may be called a parent group. By way of further example, after fetching the relevant representations of token groups through a hybrid search mechanism, using primary-secondary indexing, if the number of retrieved representations of token groups is more than a specified threshold, the context of the top K (where K is a counting number) retrieved representations of token groups may be replaced with the parent group that contains the most relevant representations of token groups. To illustrate, if relevant representations of token groups 1008 are identified in the hybrid search, through primary indexes 1004 and secondary indexes 1006, and the number of relevant representations of token groups 1008 exceeds a predetermined threshold (e.g., two representations of token groups), modeling system 102 may instead identify parent group 1010 that includes a number of most relevant representations of token groups 1008 that satisfies the predetermined threshold. For illustrative purposes only, these most relevant representations of token groups are shown in solid black in FIG. 10. Other parent groups, such as parent group 1011 , that include residual representations of token groups (e.g., shown in gray in FIG. 10), may not be returned by modeling system 102. In this manner, modeling system 102 may streamline the retrieved representations of reference data, therein reducing possible redundancy and resulting in more coherent and meaningful contextual reference data. The predetermined threshold may be set at a value to minimize redundancy in the retrieved token groups, maximize clarity in the reference data, and prevent over-inundation of too much reference data, for example.

[0155] As shown in FIG. 9, process 900 may include, at step 1012, providing retrieved representations of token groups to a generative machine learning model. For example, the most relevant parent group 1010 of representations of token groups may be provided as input to one or more generative machine learning models. The output of the one or more generative machine learning models may be used, at step 1014, to generate content that is responsive to one or more portions of the prompt that are associated with a generative prompt. If no portion of the prompt was associated with an executable action, and the output of the one or more generative machine learning models does not implicate any actions to be executed, then modeling system 102 mayprovide the generated content based on the generative output to the user in a response message. Alternatively, if a portion of the prompt was associated with an executable action and / or the output of the one or more generative machine learning models implicates one or more actions to be executed, then modeling system 102 may proceed to execute one or more actions as described herein (see, e.g., FIG. 1 1 ), which may also use the generated content.

[0156] Referring now to FIG. 1 1 , shown is a schematic diagram for a method for integrated processing of generative and instructive prompts in machine learning models, according to some non-limiting embodiments or aspects. The steps of process 1 100 shown in FIG. 1 1 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of process 1 100 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 1100 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102.

[0157] As shown in FIG. 1 1 , modeling system 102 may perform process 1 100 to construct a graph based on a user interface, determine executable actions, and execute the actions in the user interface. At step 1102, process 1100 may include extracting relevant features from the user interface. For example, modeling system 102 may extract relevant features from the user interface, such as elements and relationships between those elements (e.g., tab order, layout, attributes, dependency, etc.). Modeling system 102 may use a neural network model to extract the relevant elements and relationships.

[0158] As shown in FIG. 1 1 , at step 1 104, process 1 100 may include preprocessing the features extracted at step 1 102. For example, modeling system 102 may pre-process the extracted features to conform the features to a predetermined format, to encode the features, select out interactable elements and relevant relationships, and / or the like.

[0159] As shown in FIG. 1 1 , at step 1 106, process 1100 may include constructing a graph of the user interface. For example, modeling system 102 may construct agraph representation of the user interface, where each node of the graph is associated with an element (e.g., each node being stored with information of the attributes of that element) and each edge between two nodes is associated with the relationship between the elements that are associated with the connected nodes. To illustrate, consider a user interface that is a webpage, and the webpage includes a text box (e.g., associated with node A of the graph), a drop-down menu (e.g., associated with node B of the graph), and a submit button (e.g., associated with node C of the graph). The graph, therefore, would have three nodes. An edge from node A to node B may represent the relationship of entering text into the text box and then selecting an option from the drop-down menu based on the entered text. An edge from node B to node C may represent the relationship of clicking the submit button after making a selection from the drop-down menu. In this manner, the constructed graph may represent the expected interactivity of the user interface, for which the prompt may call for actions therein. Included or separate from step 1 106, modeling system 102 may apply a graph attention network (GAT) mechanism to the constructed graph, in step 1108. For example, modeling system 102 may use an attention mechanism to assign different importance (e.g., attention scores) to each of the neighboring nodes of a subject node.

[0160] As shown in FIG. 1 1 , at step 1 1 10, the graph that is constructed in step 1 106 may be stored in a graph datastore. For example, modeling system 102 may store the generated graph, in latent graph representation format, in a graph datastore for later traversal when executing prompted actions.

[0161] As shown in FIG. 1 1 , at step 1 1 12, process 1 100 may include receiving a prompt from a user that is in natural language format. The prompt may include one or more portions associated with a generative prompt and / or one or more portions associated with at least one executable actions. In response to receiving a prompt from a user, process 1 100 may further include, at step 11 14, translating (e.g., converting) at least a portion of the prompt from natural language into vector representation. For example, modeling system 102 may translate (e.g., convert) the prompt from natural language into vector representation using a transformer encoder model. The transformer encoder model may, by converting the natural language into vector representation, capture the intent and context of the user prompt. The vector representation may be an encoding of one or more action steps associated with, at least, one or more executable actions intended by the user’s prompt.

[0162] As shown in FIG. 1 1 , at step 11 16, process 1 100 may include performing a hybrid search of the graph datastore. For example, modeling system 102 may, using one or more vector representations of at least a portion of the prompt, perform a hybrid search (e.g., using primary-secondary indexing) of the graph datastore, to retrieve the relevant graph representation of a user interface. For example, the user’s prompt may call for sending an email, so the modeling system 102 may retrieve a graph representation of an email software application. If a graph representation does not already exist, modeling system 102 may generate a new graph representation of a user interface that is associated with the prompt received from the user.

[0163] As shown in FIG. 1 1 , at step 1 118, process 1 1 10 may include translating (e.g., converting) the vector representation into a sequence of action steps that can manipulate the user interface. For example, modeling system 102 may, using a sequence-to-sequence decoder model and input of the vector representation (e.g., encoding) of one or more action steps, convert the one or more vector representations of the one or more portions of the prompt into a sequence of action steps.

[0164] As shown in FIG. 1 1 , at step 1 120, process 1 1 10 may include performing one or more action steps to achieve the at least one executable action intended by the user’s prompt. For example, modeling system 102 may query the retrieved graph representation of the user interface by traversing the graph according to the decoded series of action steps. To continue the illustration described above in connection with FIG. 1 1 , if the sequence of actions is “Enter text”, “Select option”, and “Click submit”, the graph traversal may start at Node A (e.g., associated with the text box) to perform the action of entering text, then the graph traversal may move to Node B (e.g., associated with the drop-down menu) to perform the action of selecting an option, and then the graph traversal may move to Node C (e.g., associated with the submit button) to perform the action of activating (e.g., clicking) the submit button.

[0165] As shown in FIG. 1 1 , at step 1 122, process 1 1 10 may include providing a response to the user based on the performed action steps. For example, modeling system 102 may provide responsive feedback based on the actions that have been performed. To continue the foregoing illustration, after the submit button has been activated, the user interface may be caused to be updated, at least in part caused by modeling system 102. The response to the user may further include displaying a message (e.g., in pop-up window, frame, field, etc.) in the user interface to briefly describe the actions that were executed. The complexity and substance of theresponse to the user may be associated with the complexity and substance of the action steps executed by modeling system 102.

[0166] Although embodiments have been described in detail for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect. In fact, any of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

Claims

WHAT IS CLAIMED IS:1 . A system comprising: at least one processor configured to: receive reference data; store a representation of the reference data in a datastore; receive a prompt; determine a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action; retrieve a subset of the representation of the reference data from the datastore based on the first portion of the prompt; determine a generative output from a machine learning model based on an input comprising (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt; generate content responsive to the first portion of the prompt based on the generative output; determine an encoding of a plurality of action steps based on the second portion of the prompt; and execute the at least one executable action using a sequence-to- sequence decoder model and based on the content and the encoding of the plurality of action steps.

2. The system of claim 1 , wherein the reference data comprises batch data and stream data, and wherein, when receiving the reference data, the at least one processor is further configured to receive the batch data and the stream data in separate threads of a multi-threaded communication channel.

3. The system of claim 1 , wherein the at least one processor is further configured to, in response to receiving the reference data: pre-process the reference data to conform to a predetermined data format, to produce formatted reference data; and dispatch the formatted reference data to at least one queue.

4. The system of claim 3, wherein the at least one processor is further configured to, in response to the formatted reference data being dispatched to the at least one queue: retrieve, in queue order, the formatted reference data from the at least one queue; generate a plurality of tokens based on the formatted reference data; generate a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups; and convert the plurality of token groups into a plurality of representations using a distributed embedding model.

5. The system of claim 4, wherein the representation of the reference data comprises the plurality of representations, and wherein, when storing the representation of the reference data, the at least one processor is further configured to store the plurality of representations in the datastore.

6. The system of claim 4, wherein the distributed embedding model is executed at least partly on a server cluster comprising at least one primary node and a plurality of secondary nodes, and wherein, when converting the plurality of token groups into the plurality of representations using the distributed embedding model, the at least one processor is further configured to, for each token group of the plurality of token groups: transmit the token group to the primary node; determine a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group; direct the token group to the corresponding secondary node from the primary node; and generate a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

7. The system of claim 6, wherein each secondary node of the plurality of secondary nodes comprises a complete set of layers of the distributedembedding model, and wherein, when determining the corresponding secondary node, the at least one processor is further configured to determine the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

8. The system of claim 1 , wherein, when determining the encoding of the plurality of action steps based on the second portion of the prompt, the at least one processor is further configured to: convert the second portion of the prompt into a vector representation using a transformer encoder model; and determine the encoding of the plurality of action steps using a sequence- to-sequence encoder model and based on the vector representation.

9. The system of claim 1 , wherein, when receiving the prompt, the at least one processor is further configured to receive the prompt via a user interface of a computing device of a user; and wherein the at least one processor is further configured to: extract a plurality of features based on the user interface using a neural network model, the plurality of features associated with (i) a plurality of elements of the user interface and (ii) a plurality of relationships between the plurality of elements; and construct a graph of the user interface based on the plurality of features, the graph comprising a plurality of nodes and a plurality of edges, each node of the plurality of nodes associated with an element of the plurality of elements, and each edge of the plurality of edges associated with a relationship of the plurality of relationships.

10. The system of claim 9, wherein, when executing the at least one executable action, the at least one processor is further configured to traverse the graph of the user interface using the sequence-to-sequence decoder model and based on the encoding of the plurality of action steps.1 1. A computer-implemented method comprising: receiving, with at least one processor, reference data;storing, with at least one processor, a representation of the reference data in a datastore; receiving, with at least one processor, a prompt; determining, with at least one processor, a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action; retrieving, with at least one processor, a subset of the representation of the reference data from the datastore based on the first portion of the prompt; determining, with at least one processor, a generative output from a machine learning model based on an input comprising (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt; generating, with at least one processor, content responsive to the first portion of the prompt based on the generative output; determining, with at least one processor, an encoding of a plurality of action steps based on the second portion of the prompt; and executing, with at least one processor, the at least one executable action using a sequence-to-sequence decoder model and based on the content and the encoding of the plurality of action steps.

12. The method of claim 1 1 , further comprising, in response to receiving the reference data: pre-processing, with at least one processor, the reference data to conform to a predetermined data format, to produce formatted reference data; and dispatching, with at least one processor, the formatted reference data to at least one queue.

13. The method of claim 12, further comprising, in response to the formatted reference data being dispatched to the at least one queue: retrieving, with at least one processor and in queue order, the formatted reference data from the at least one queue; generating, with at least one processor, a plurality of tokens based on the formatted reference data;generating, with at least one processor, a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups; and converting, with at least one processor, the plurality of token groups into a plurality of representations using a distributed embedding model.

14. The method of claim 13, wherein the distributed embedding model is executed at least partly on a server cluster comprising at least one primary node and a plurality of secondary nodes, and wherein converting the plurality of token groups into the plurality of representations using the distributed embedding model further comprises, for each token group of the plurality of token groups: transmitting the token group to the primary node; determining a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group; directing the token group to the corresponding secondary node from the primary node; and generating a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

15. The method of claim 14, wherein each secondary node of the plurality of secondary nodes comprises a complete set of layers of the distributed embedding model, and wherein determining the corresponding secondary node further comprises determining the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

16. A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: receive reference data; store a representation of the reference data in a datastore; receive a prompt;determine a first portion of the prompt associated with at least one generative prompt, and a second portion of the prompt associated with at least one executable action; retrieve a subset of the representation of the reference data from the datastore based on the first portion of the prompt; determine a generative output from a machine learning model based on an input comprising (i) the subset of the representation of the reference data and (ii) at least part of the first portion of the prompt; generate content responsive to the first portion of the prompt based on the generative output; determine an encoding of a plurality of action steps based on the second portion of the prompt; and execute the at least one executable action using a sequence-to- sequence decoder model and based on the content and the encoding of the plurality of action steps.

17. The computer program product of claim 16, wherein the program instructions further cause the at least one processor to, in response to receiving the reference data: pre-process the reference data to conform to a predetermined data format, to produce formatted reference data; and dispatch the formatted reference data to at least one queue.

18. The computer program product of claim 17, wherein the program instructions further cause the at least one processor to, in response to the formatted reference data being dispatched to the at least one queue: retrieve, in queue order, the formatted reference data from the at least one queue; generate a plurality of tokens based on the formatted reference data; generate a plurality of token groups based on the plurality of tokens, each token group of the plurality of token groups sharing at least one token with another token group of the plurality of token groups; and convert the plurality of token groups into a plurality of representations using a distributed embedding model.

19. The computer program product of claim 18, wherein the distributed embedding model is executed at least partly on a server cluster comprising at least one primary node and a plurality of secondary nodes, and wherein the program instructions that cause the at least one processor to convert the plurality of token groups into the plurality of representations using the distributed embedding model further cause the at least one processor to, for each token group of the plurality of token groups: transmit the token group to the primary node; determine a corresponding secondary node of the plurality of secondary nodes based at least partly on a data modality of the token group; direct the token group to the corresponding secondary node from the primary node; and generate a representation of the token group by executing at least one layer of the distributed embedding model on the corresponding secondary node of the server cluster.

20. The computer program product of claim 19, wherein each secondary node of the plurality of secondary nodes comprises a complete set of layers of the distributed embedding model, and wherein the program instructions that cause the at least one processor to determine the corresponding secondary node further cause the at least one processor to determine the corresponding secondary node of the plurality of secondary nodes based at least partly on a load balance of the server cluster.

Images