System, method, and computer program product for reducing order bias of machine learning models via feature re-ordering

Abstract

Systems, methods, and computer program products for reducing order bias of machine learning models via feature re-ordering are provided. An example system may include a processor to receive a training dataset comprising a plurality of data instances for a plurality of features, generate a feature trajectory of the plurality of features based on an output of a critic machine learning model, wherein the output of the critic machine learning model comprises a prediction of a value of loss of an output of a large language model (LLM), estimate a value of loss of an output of the LLM based on the feature trajectory to provide an estimated value of loss, update one or more parameters of the critic machine learning model to minimize the estimated value to provide an updated critic machine learning model, and generate a final output using the updated critic machine learning model.

Description

Attorney Docket No. 08223-2503018 (9604W001)SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR REDUCING ORDER BIAS OF MACHINE LEARNING MODELS VIA FEATURE RE-ORDERINGCROSS REFERENCE TO RELATED APPLICATION

[0001] The present application claims the benefit of United States Provisional Patent Application No. 63 / 701 ,788, filed on October 1 , 2024, the disclosure of which is hereby incorporated by reference in its entirety.BACKGROUND1 . Technical Field

[0002] This disclosure relates generally to applications of machine learning models and, in non-limiting embodiments or aspects, to systems, methods, and computer program products for reducing order bias of machine learning models via feature reordering.2. Technical Considerations

[0003] Machine learning is a field of computer science that may use statistical techniques to provide a computer system with the ability to learn (e.g., to progressively improve performance of) a task with data without the computer system being explicitly programmed to perform the task. In some instances, machine learning models, such as neural network machine learning models, may be developed for sets of data so that the machine learning models can perform a task (e.g., a task associated with a prediction) with regard to the set of data.

[0004] A large language model (LLM) may refer to a computational model capable of language generation or other natural language processing tasks. As language models, LLMs may acquire these abilities by learning statistical relationships from vast amounts of text during a self-supervised and / or semi-supervised training process. In some instances, an LLM may include an artificial neural network architecture built with a decoder-only transformer-based architecture, which may enable efficient processing and generation of large-scale text data. Further, an LLM can be fine-tuned for specific tasks and / or can be guided by prompt engineering. An LLM may acquire predictive power regarding syntax, semantics, and / or ontologies inherent in human language corpora.67P3321.DOCX Page 1 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0005] However, an LLM may also inherit inaccuracies and / or biases present in the data upon which the LLM was trained. In some instances, order bias may exist in machine learning models, including foundational models (e.g., which may include LLMs). Order bias may refer to a problem where a prediction result of a machine learning model may be biased according to feature order. Such order bias may be caused by order of data instances (e.g., order of features in data instances) of training data.SUMMARY

[0006] Accordingly, provided are improved systems, methods, and computer program products for reducing order bias of machine learning models via feature reordering.

[0007] According to non-limiting embodiments or aspects, provided is a system for reducing order bias of machine learning models via feature re-ordering, may include at least one processor configured to: receive a training dataset comprising a plurality of data instances for a plurality of features; generate a feature trajectory based on the plurality of features, wherein the feature trajectory may include a predicted ordered set of values of at least a subset of features of the plurality of features; estimate a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model may include a prediction of a value of loss of an output of the LLM based on an input, and wherein the input may include the feature trajectory; update one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generate a final output using the updated critic machine learning model, wherein the final output may include a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

[0008] In some non-limiting embodiments or aspects, when receiving the training dataset comprising the plurality of data instances for the plurality of features, the at least one processor is configured to: receive an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.67P3321.DOCX Page 2 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0009] In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, wherein the final output of the LLM is based on an input, wherein the input may include the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features may include an ordered set of values of all of the features of the plurality of features.

[0010] In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input may include the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features may include an ordered set of values of a subset of features of the plurality of features.

[0011] In some non-limiting embodiments or aspects, when updating the one or more parameters of the critic machine learning model, the at least one processor is configured to: update one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

[0012] In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

[0013] In some non-limiting embodiments or aspects, when generating a final output using the updated critic machine learning model, the at least one processor is configured to: generate a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output may include a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the at least one processor is further configured to: provide the final output as an input to the67P3321.DOCX Page 3 of 44Attorney Docket No. 08223-2503018 (9604W001)LLM; generate a result of the LLM based on the input; and perform an action based on the result of the LLM.

[0014] According to non-limiting embodiments or aspects, provided is a computer implemented method for reducing order bias of machine learning models via feature re-ordering, may include receiving, with at least one processor, a training dataset comprising a plurality of data instances for a plurality of features; generating, with at least one processor, a feature trajectory based on the plurality of features, wherein the feature trajectory may include a predicted ordered set of values of at least a subset of features of the plurality of features; estimating, with at least one processor, a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model may include a prediction of a value of loss of an output of the LLM based on an input, and wherein the input may include the feature trajectory; updating, with at least one processor, one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generating, with at least one processor, a final output using the updated critic machine learning model, wherein the final output may include a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

[0015] In some non-limiting embodiments or aspects, receiving the training dataset comprising the plurality of data instances for the plurality of features may include: receiving an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.

[0016] In some non-limiting embodiments or aspects, estimating a value of loss of an output of the LLM may include: estimating a value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, wherein the final output of the LLM is based on an input, wherein the input may include the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features may include an ordered set of values of all of the features of the plurality of features.67P3321.DOCX Page 4 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0017] In some non-limiting embodiments or aspects, estimating a value of loss of an output of the LLM may include: estimating a value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input may include the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features may include an ordered set of values of a subset of features of the plurality of features.

[0018] In some non-limiting embodiments or aspects, updating the one or more parameters of the critic machine learning model may include: updating one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

[0019] In some non-limiting embodiments or aspects, estimating a value of loss of an output of the LLM may include: estimating a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

[0020] In some non-limiting embodiments or aspects, generating a final output using the updated critic machine learning model may include: generating a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output may include a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the method further may include: providing the final output as an input to the LLM; generating a result of the LLM based on the input; and performing an action based on the result of the LLM.

[0021] According to non-limiting embodiments or aspects, provided is a computer program product for reducing order bias of machine learning models via feature reordering, comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, causes the at least one processor to: receive a training dataset comprising a plurality of data instances for a plurality of features; generate a feature trajectory based on the plurality of features, wherein the feature trajectory may include a predicted ordered set of values of at least a subset of features of the plurality of features; estimate a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an67P3321.DOCX Page 5 of 44Attorney Docket No. 08223-2503018 (9604W001) estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model may include a prediction of a value of loss of an output of the LLM based on an input, and wherein the input may include the feature trajectory; update one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generate a final output using the updated critic machine learning model, wherein the final output may include a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

[0022] In some non-limiting embodiments or aspects, the program instructions that cause the at least one processor to receive the training dataset comprising the plurality of data instances for the plurality of features, cause the at least one processor to: receive an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.

[0023] In some non-limiting embodiments or aspects, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, wherein the final output of the LLM is based on an input, wherein the input may include the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features may include an ordered set of values of all of the features of the plurality of features.

[0024] In some non-limiting embodiments or aspects, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input may include the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features may include an ordered set of values of a subset of features of the plurality of features.

[0025] In some non-limiting embodiments or aspects, the program instructions that cause the at least one processor to update the one or more parameters of the critic machine learning model, cause the at least one processor to: update one or more67P3321.DOCX Page 6 of 44Attorney Docket No. 08223-2503018 (9604W001) parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

[0026] In some non-limiting embodiments or aspects, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

[0027] In some non-limiting embodiments or aspects, the program instructions that cause the at least one processor to generate a final output using the updated critic machine learning model, cause the at least one processor to: generate a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output may include a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the program instructions further cause the at least one processor to: provide the final output as an input to the LLM; generate a result of the LLM based on the input; and perform an action based on the result of the LLM.

[0028] Further non-limiting embodiments or aspects will be set forth in the following numbered clauses:

[0029] Clause 1 : A system, comprising: at least one processor configured to: receive a training dataset comprising a plurality of data instances for a plurality of features; generate a feature trajectory based on the plurality of features, wherein the feature trajectory comprises a predicted ordered set of values of at least a subset of features of the plurality of features; estimate a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model comprises a prediction of a value of loss of an output of the LLM based on an input, and wherein the input comprises the feature trajectory; update one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generate a final output using the updated critic machine learning model, wherein the67P3321.DOCX Page 7 of 44Attorney Docket No. 08223-2503018 (9604W001) final output comprises a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

[0030] Clause 2: The system of clause 1 , wherein, when receiving the training dataset comprising the plurality of data instances for the plurality of features, the at least one processor is configured to: receive an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.

[0031] Clause 3: The system of clause 1 or 2, wherein, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, wherein the final output of the LLM is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of all of the features of the plurality of features.

[0032] Clause 4: The system of any of clauses 1 -3, wherein, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of a subset of features of the plurality of features.

[0033] Clause 5: The system of any of clauses 1 -4, wherein, when updating the one or more parameters of the critic machine learning model, the at least one processor is configured to: update one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

[0034] Clause 6: The system of any of clauses 1 -5, wherein, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.67P3321.DOCX Page 8 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0035] Clause 7: The system of any of clauses 1 -6, wherein, when generating a final output using the updated critic machine learning model, the at least one processor is configured to: generate a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output comprises a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the at least one processor is further configured to: provide the final output as an input to the LLM; generate a result of the LLM based on the input; and perform an action based on the result of the LLM.

[0036] Clause 8: A computer-implemented method, comprising: receiving, with at least one processor, a training dataset comprising a plurality of data instances for a plurality of features; generating, with at least one processor, a feature trajectory based on the plurality of features, wherein the feature trajectory comprises a predicted ordered set of values of at least a subset of features of the plurality of features; estimating, with at least one processor, a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model comprises a prediction of a value of loss of an output of the LLM based on an input, and wherein the input comprises the feature trajectory; updating, with at least one processor, one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generating, with at least one processor, a final output using the updated critic machine learning model, wherein the final output comprises a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

[0037] Clause 9: The computer-implemented method of clause 8, wherein receiving the training dataset comprising the plurality of data instances for the plurality of features comprises: receiving an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.

[0038] Clause 10: The computer-implemented method of clause 8 or 9, wherein estimating a value of loss of an output of the LLM comprises: estimating a value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide67P3321.DOCX Page 9 of 44Attorney Docket No. 08223-2503018 (9604W001) an estimated value of loss of the output of the LLM, wherein the final output of the LLM is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of all of the features of the plurality of features.

[0039] Clause 1 1 : The computer-implemented method of any of clauses 8-10, wherein estimating a value of loss of an output of the LLM comprises: estimating a value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of a subset of features of the plurality of features.

[0040] Clause 12: The computer-implemented method of any of clauses 8-1 1 , wherein updating the one or more parameters of the critic machine learning model comprises: updating one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

[0041] Clause 13: The computer-implemented method of any of clauses 8-12, wherein estimating a value of loss of an output of the LLM comprises: estimating a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

[0042] Clause 14: The computer-implemented method of any of clauses 8-13, wherein generating a final output using the updated critic machine learning model comprises: generating a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output comprises a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the method further comprises: providing the final output as an input to the LLM; generating a result of the LLM based on the input; and performing an action based on the result of the LLM.

[0043] Clause 15: A computer program product, comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, causes the at least one processor to: at least one67P3321.DOCX Page 10 of 44Attorney Docket No. 08223-2503018 (9604W001) processor configured to: receive a training dataset comprising a plurality of data instances for a plurality of features; generate a feature trajectory based on the plurality of features, wherein the feature trajectory comprises a predicted ordered set of values of at least a subset of features of the plurality of features; estimate a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model comprises a prediction of a value of loss of an output of the LLM based on an input, and wherein the input comprises the feature trajectory; update one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generate a final output using the updated critic machine learning model, wherein the final output comprises a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

[0044] Clause 16: The computer program product of clause 15, wherein, the program instructions that cause the at least one processor to receive the training dataset comprising the plurality of data instances for the plurality of features, cause the at least one processor to: receive an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.

[0045] Clause 17: The computer program product of clause 15 or 16, wherein, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, wherein the final output of the LLM is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of all of the features of the plurality of features.

[0046] Clause 18: The computer program product of any of clauses 15-17, wherein, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic67P3321.DOCX Page 1 1 of 44Attorney Docket No. 08223-2503018 (9604W001) machine learning model is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of a subset of features of the plurality of features.

[0047] Clause 19: The computer program product of any of clauses 15-18, wherein, the program instructions that cause the at least one processor to update the one or more parameters of the critic machine learning model, cause the at least one processor to: update one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

[0048] Clause 20: The computer program product of any of clauses 15-19, wherein, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

[0049] Clause 21 : The computer program product of any of clauses 15-20, wherein, the program instructions that cause the at least one processor to generate a final output using the updated critic machine learning model, cause the at least one processor to: generate a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output comprises a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the program instructions further cause the at least one processor to: provide the final output as an input to the LLM; generate a result of the LLM based on the input; and perform an action based on the result of the LLM.

[0050] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of67P3321.DOCX Page 12 of 44Attorney Docket No. 08223-2503018 (9604W001) illustration and description only and are not intended as a definition of the limits of the disclosed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Additional advantages and details are explained in greater detail below with reference to the non-limiting, exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0052] FIG. 1 is a schematic diagram of a system for reducing order bias of machine learning models via feature re-ordering, according to some non-limiting embodiments or aspects;

[0053] FIG. 2 is a flow diagram for a process for reducing order bias of machine learning models via feature re-ordering, according to some non-limiting embodiments or aspects;

[0054] FIGS. 3A-3E are schematic diagrams of an exemplary implementation of a process for reducing order bias of machine learning models via feature re-ordering, according to some non-limiting embodiments or aspects;

[0055] FIG. 4 is a diagram of an exemplary environment in which systems, methods, and / or computer program products, described herein, may be implemented, according to some non-limiting embodiments or aspects; and

[0056] FIG. 5 is a schematic diagram of example components of one or more devices of FIG. 1 and / or FIG. 4, according to some non-limiting embodiments or aspects.DETAILED DESCRIPTION

[0057] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to the embodiments as they are oriented in the drawing figures. However, it is to be understood that the embodiments may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary embodiments or aspects of the disclosed subject matter. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.67P3321.DOCX Page 13 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0058] Some non-limiting embodiments or aspects may be described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0059] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, and / or the like) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. In addition, reference to an action being “based on” a condition may refer to the action being “in response to” the condition. For example, the phrases “based on” and “in response to” may, in some non-limiting embodiments or aspects, refer to a condition for automatically triggering an action (e.g., a specific operation of an electronic device, such as a computing device, a processor, and / or the like).

[0060] As used herein, the term “acquirer institution” may refer to an entity licensed and / or approved by a transaction service provider to originate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. The transactions the acquirer institution may originate may include payment transactions (e.g., purchases, original credit transactions (OCTs), account funding transactions (AFTs), and / or the like). In some non-limiting embodiments or aspects, an acquirer institution may be a financial institution, such as a bank. As used herein, the term “acquirer system” may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0061] As used herein, the term “account identifier” may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a67P3321.DOCX Page 14 of 44Attorney Docket No. 08223-2503018 (9604W001) customer account. The term “token” may refer to an identifier that is used as a substitute or replacement identifier for an original account identifier, such as a PAN. Account identifiers may be alphanumeric or any combination of characters and / or symbols. Tokens may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, and / or the like) such that they may be used to conduct a transaction without directly using the original account identifier. In some examples, an original account identifier, such as a PAN, may be associated with a plurality of tokens for different individuals or purposes.

[0062] As used herein, the term “communication” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of data (e.g., information, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second units. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit.

[0063] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.67P3321.DOCX Page 15 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0064] As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the Internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.”

[0065] As used herein, the term “system” may refer to one or more computing devices or combinations of computing devices (e.g., processors, servers, client devices, software applications, components of such, and / or the like). Reference to “a device,” “a server,” “a processor,” and / or the like, as used herein, may refer to a previously-recited device, server, or processor that is recited as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and the claims, a first device, a first server, or a first processor that is recited as performing a first step or a first function may refer to the same or different device, server, or processor recited as performing a second step or a second function.

[0066] As used herein, the term “issuer institution” may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide an account identifier, such as a PAN, to a customer that uniquely identifies one or more accounts associated with that customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument, e.g., a payment card, and / or may be electronic and used for electronic payments. The term “issuer system” refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing a transaction.

[0067] As used herein, the term “merchant” may refer to an individual or entity that provides goods and / or services, or access to goods and / or services, to customers based on a transaction, such as a payment transaction. The term “merchant” or “merchant system” may also refer to one or more computer systems operated by or67P3321.DOCX Page 16 of 44Attorney Docket No. 08223-2503018 (9604W001) on behalf of a merchant, such as a server computer executing one or more software applications.

[0068] As used herein, the term “payment device” may refer to an electronic payment device, a portable financial device (e.g., a payment card, such as a credit or debit card), a gift card, a smartcard, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a keychain device or fob, a radio frequency identification (RFID) transponder, a retailer discount or loyalty card, a cellular phone, an electronic wallet mobile application, a PDA, a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, and / or the like. In some non-limiting embodiments or aspects, the payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, a name of the account holder, and / or the like).

[0069] As used herein, a “point-of-sale (POS) device” may refer to one or more devices, which may be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include peripheral devices, card readers, scanning devices (e.g., code scanners), Bluetooth® communication receivers, near-field communication (NFC) receivers, RFID receivers, and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, and / or the like. As used herein, a “point-of-sale (POS) system” may refer to one or more client devices and / or peripheral devices used by a merchant to conduct a transaction. For example, a POS system may include one or more POS devices and / or other like devices that may be used to conduct a payment transaction. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers configured to process online payment transactions through webpages, mobile applications, and / or the like.

[0070] As used herein, the term “transaction service provider” may refer to an entity that receives transaction authorization requests from merchants or other entities and provides guarantees of payment, in some cases through an agreement between the transaction service provider and an issuer institution. For example, a transaction service provider may include a payment network such as Visa® or any other entity that processes transactions. The term “transaction processing system” may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software67P3321.DOCX Page 17 of 44Attorney Docket No. 08223-2503018 (9604W001) applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0071] Non-limiting embodiments or aspects of the disclosed subject matter are directed to systems, methods, and computer program products for reducing order bias of machine learning models via feature re-ordering. The disclosed subject matter provides a machine learning (ML) optimization system that is configured to receive a training dataset comprising a plurality of data instances for a plurality of features, generate a feature trajectory based on the plurality of features, where the feature trajectory comprises a predicted ordered set of values of at least a subset of features of the plurality of features, estimate a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, where the output of the critic machine learning model comprises a prediction of a value of loss of an output of the LLM based on an input, where the input comprises the feature trajectory, update one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model, and generate a final output using the updated critic machine learning model, where the final output comprises a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

[0072] In some non-limiting embodiments or aspects, when receiving the training dataset comprising the plurality of data instances for the plurality of features, the ML optimization system may receive an ordered set of values of the plurality of features for each data instance, where the ordered set of values is based on a number of features included in the training dataset.

[0073] In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, the ML optimization system may estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, where the final output of the LLM is based on an input, where the input comprises the feature trajectory of the plurality of features, and where the feature trajectory of the plurality of features comprises an ordered set of values of all of the features of the plurality of features.67P3321.DOCX Page 18 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0074] In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, the ML optimization system may estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of a subset of features of the plurality of features.

[0075] In some non-limiting embodiments or aspects, when updating the one or more parameters of the critic machine learning model, the ML optimization system may update one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

[0076] In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, the ML optimization system may estimate a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

[0077] In some non-limiting embodiments or aspects, when generating a final output using the updated critic machine learning model, the ML optimization system may generate a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output comprises a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM, and the ML optimization system may provide the final output as an input to the LLM, generate a result of the LLM based on the input, and perform an action based on the result of the LLM.

[0078] In this way, the disclosed subject matter may provide for reducing or removing the effects of order bias on a prediction result of a machine learning model models may be biased according to feature order using feature trajectory. For example, the disclosed subject matter may provide for aligning data instances with an order bias (e.g., feature order bias) of an LLM. In some non-limiting embodiments or aspects, feature trajectory may pertain to values, importance, and / or representation changes over time, that correspond to training steps and / or layers of a machine learning model, which may be used to track the evolution of features, instead of solely tracking static values of features. Additionally, the disclosed subject matter may67P3321.DOCX Page 19 of 44Attorney Docket No. 08223-2503018 (9604W001) provide for a machine learning model, such as an MLP, that has been generated to reduce or remove the effects of order bias on a prediction result of a LLM based on the use of a critic machine learning model and that may provide data that is to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM. Accordingly, the disclosed subject matter is unique and unconventional.

[0079] For the purpose of illustration, in the following description, while the presently disclosed subject matter is described with respect to systems, methods, and computer program products for reducing order bias of machine learning models via feature reordering, one skilled in the art will recognize that the disclosed subject matter is not limited to the non-limiting embodiments or aspects disclosed herein. For example, the systems, methods, and computer program products, described herein, may be used with a wide variety of settings, such as machine learning models, including LLMs, that are used for making determinations (e.g., predictions, classifications, regressions, and / or the like) based on a dataset, such as for fraud detection / prevention, authorization, authentication, identification, product recommendation, and / or the like.

[0080] Referring now to FIG. 1 , shown is example system 100 for reducing order bias of machine learning models via feature re-ordering, according to some nonlimiting embodiments or aspects. For example, system 100 may include machine learning (ML) optimization system 102, ML model database 104, user device 106, and / or communication network 108.

[0081] ML optimization system 102 may include one or more devices capable of receiving information from and / or communicating information to ML model database 104 and / or user device 106 (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, and / or the like). For example, ML optimization system 102 may include a computing device, such as a server, a group of servers, a desktop computer, a portable computer, a mobile device, and / or other like devices. In some non-limiting embodiments or aspects, ML optimization system 102 may be in communication with a data storage device (e.g., ML model database 104), which may be local or remote to ML optimization system 102. In some non-limiting embodiments or aspects, ML optimization system 102 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device (e.g., ML model database 104).67P3321.DOCX Page 20 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0082] ML model database 104 may include one or more devices capable of receiving information from and / or communicating information to ML optimization system 102 and / or user device 106 (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, and / or the like). For example, ML model database 104 may include a computing device, such as a server, a group of servers, a desktop computer, a portable computer, a mobile device, and / or other like devices. In some non-limiting embodiments or aspects, ML model database 104 may include a data storage device. In some non-limiting embodiments or aspects, ML model database 104 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device. In some non-limiting embodiments or aspects, ML model database 104 may be part of ML optimization system 102 and / or part of the same system as ML optimization system 102.

[0083] User device 106 may include one or more devices capable of receiving information from and / or communicating information to ML optimization system 102 and / or ML model database 104 (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, and / or the like). For example, user device 106 may include a computing device, such as a mobile device, a portable computer, a desktop computer, and / or other like devices. Additionally or alternatively, each user device 106 may include a device capable of receiving information from and / or communicating information to other user devices 106 (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, and / or the like). In some non-limiting embodiments or aspects, user device 106 may be part of ML optimization system 102 and / or part of the same system as ML optimization system 102. For example, ML optimization system 102, ML model database 104, and user device 106 may all be (and / or be part of) a single system and / or a single computing device.

[0084] Communication network 108 may include one or more wired and / or wireless networks. For example, communication network 108 may include a cellular network (e.g., a long-term evolution (LTE®) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, and / or the like), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private67P3321.DOCX Page 21 of 44Attorney Docket No. 08223-2503018 (9604W001) network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0085] The number and arrangement of systems and devices shown in FIG. 1 are provided as an example. There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or differently arranged systems and / or devices than those shown in FIG. 1. Furthermore, two or more systems or devices shown in FIG. 1 may be implemented within a single system or device, or a single system or device shown in FIG. 1 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another set of systems or another set of devices of system 100.

[0086] Referring now to FIG. 2, shown is a flow diagram for process 200 for reducing order bias of machine learning models via feature re-ordering, according to some non-limiting embodiments or aspects. The steps shown in FIG. 2 are for example purposes only. It will be appreciated that additional, fewer, different, and / or different order of steps may be used in non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, process 200 may be performed during a training process. In some non-limiting embodiments or aspects, one or more of the steps of process 200 may be performed (e.g., completely, partially, and / or the like) by ML optimization system 102 (e.g., at least one computing device of ML optimization system 102). In some non-limiting embodiments or aspects, one or more of the steps of process 200 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including ML optimization system 102, such as ML model database 104, user device 106, and / or the like.

[0087] As shown in FIG. 2, at step 202, process 200 may include receiving a training dataset that includes a plurality of data instances for a plurality of features. For example, ML optimization system 102 may receive the training dataset. In some nonlimiting embodiments or aspects, when receiving the training dataset comprising the plurality of data instances for the plurality of features, ML optimization system 102 may67P3321.DOCX Page 22 of 44Attorney Docket No. 08223-2503018 (9604W001) receive an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.

[0088] In some non-limiting embodiments or aspects, the dataset may include a plurality of data instances, the plurality of data instances including a plurality of features. In some non-limiting embodiments or aspects, the plurality of features may include a first plurality of numerical features and a second plurality of categorical features. In some non-limiting embodiments or aspects, the dataset may include tabular data. In some non-limiting embodiments or aspects, ML optimization system 102 may perform a serialization procedure (e.g., a text-based serialization procedure) to convert tabular data of a dataset into text data to fit an input modality of an LLM.

[0089] In some non-limiting embodiments or aspects, one or more features of the plurality of features may correspond to one or more labels of a plurality of labels associated with the classification task. In some examples, the plurality of features may include a large number of features, such as 100 features, 500 features, 1 ,000 features, 5,000 features, 10,000 features, 25,000 features, 50,000 features, 100,000 features, 1 ,000,000 features, and / or the like. In some non-limiting embodiments or aspects, the dataset may be associated with a population of entities (e.g., consumers, requesting systems, users, accountholders, merchants, issuers, etc.) and includes a plurality of data instances associated with the plurality of features (e.g., a plurality of values of features that are to be provided as an input, which may be called input data, to a machine learning model). In some non-limiting embodiments or aspects, the plurality of data instances may represent a plurality of interactions (e.g., transactions, such as electronic payment transactions) conducted that involve the population. In some examples, the data may include a large amount of data instances, such as 100 data instances, 500 data instances, 1 ,000 data instances, 5,000 data instances, 10,000 data instances, 25,000 data instances, 50,000 data instances, 100,000 data instances, 1 ,000,000 data instances, and / or the like.

[0090] In some non-limiting embodiments or aspects, each data instance may include transaction data associated with the transaction. In some non-limiting embodiments or aspects, the transaction data may include a plurality of transaction parameters associated with an electronic payment transaction. In some non-limiting embodiments or aspects, a plurality of features may represent the plurality of transaction parameters. In some non-limiting embodiments or aspects, the plurality of67P3321.DOCX Page 23 of 44Attorney Docket No. 08223-2503018 (9604W001) transaction parameters may include electronic wallet card data associated with an electronic card (e.g., an electronic credit card, an electronic debit card, an electronic loyalty card, and / or the like), decision data associated with a decision (e.g., a decision to approve or deny a transaction authorization request), authorization data associated with an authorization response (e.g., an approved spending limit, an approved transaction value, and / or the like), a PAN, an authorization code (e.g., a personal identification number (PIN), etc.), data associated with a transaction amount (e.g., an approved limit, a transaction value, etc.), data associated with a transaction date and time, data associated with a conversion rate of a currency, data associated with a merchant type (e.g., a merchant category code that indicates a type of goods, such as grocery, fuel, and / or the like), data associated with an acquiring institution country, data associated with an identifier of a country associated with the PAN, data associated with a response code, data associated with a merchant identifier (e.g., a merchant name, a merchant location, and / or the like), data associated with a type of currency corresponding to funds stored in association with the PAN, and / or the like.

[0091] As shown in FIG. 2, at step 204, process 200 may include generating a feature trajectory based on the plurality of features. For example, ML optimization system 102 may generate a feature trajectory based on the plurality of features. The feature trajectory may include a predicted ordered set of values of at least a subset of features of the plurality of features. In some non-limiting embodiments or aspects, a value of a feature trajectory may be defined as a value (e.g., a potential maximal value) of a feature in a next (e.g., subsequent) state. In some non-limiting embodiments or aspects, values of a feature trajectory may include a sequence of values that a specific feature takes over time.

[0092] In some non-limiting embodiments or aspects, ML optimization system 102 may generate a feature trajectory based on a prompt that is to be provided to an LLM. In some non-limiting embodiments or aspects, the plurality of features may correspond to (e.g., an order of) the characters (e.g., words, numbers, symbols, etc.) included in the prompt. For example, the plurality of features may correspond to the words (e.g., words, numbers, symbols, etc.) included in a text prompt that is seeking an answer from an LLM.

[0093] In some non-limiting embodiments or aspects, ML optimization system 102 may generate the feature trajectory based on the plurality of features using a critic machine learning model (e.g., a meta-controller). For example, ML optimization67P3321.DOCX Page 24 of 44Attorney Docket No. 08223-2503018 (9604W001) system 102 may input the plurality of features to the critic machine learning model and the critic machine learning model may generate an output based on the input. The output may include a feature trajectory for the plurality of features. In some nonlimiting embodiments or aspects, ML optimization system 102 may generate the feature trajectory auto-regressively using the critic machine learning model. In some non-limiting embodiments or aspects, the critic machine learning model may be configured to estimate value functions of feature trajectories through dynamic programming. The value function of a specific feature trajectory may align with the performance of an LLM based on a serialized feature input in a particular order (e.g., which may be used to supervise training of the critic machine learning model).

[0094] In some non-limiting embodiments or aspects, the critic machine learning model may include a plurality of layers and / or another machine learning model. For example, the critic machine learning model may include an MLP machine learning model.

[0095] As shown in FIG. 2, at step 206, process 200 may include estimating a value of loss of an output of an LLM based on a critic machine learning model. For example, ML optimization system 102 may estimate a value of loss of the output of the LLM based on the critic machine learning model. In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, ML optimization system 102 may estimate an MSE value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

[0096] In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, ML optimization system 102 may estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input that includes the feature trajectory of the plurality of features, and the feature trajectory of the plurality of features may include an ordered set of values of a subset of features of the plurality of features.

[0097] In some non-limiting embodiments or aspects, when estimating a value of loss of an output of the LLM, ML optimization system 102 may estimate a value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, wherein the final output of the LLM is67P3321.DOCX Page 25 of 44Attorney Docket No. 08223-2503018 (9604W001) based on an input that includes the feature trajectory of the plurality of features, and the feature trajectory of the plurality of features may include an ordered set of values of all of the features of the plurality of features.

[0098] As shown in FIG. 2, at step 208, process 200 may include updating one or more parameters of the critic machine learning model. For example, ML optimization system 102 may update one or more parameters of the critic machine learning model to provide an updated critic machine learning model. In some non-limiting embodiments or aspects, when updating the one or more parameters of the critic machine learning model, ML optimization system 102 may update one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM.

[0099] As shown in FIG. 2, at step 210, process 200 may include generating a final output using an updated critic machine learning model. For example, ML optimization system 102 may generate a final output using the updated critic machine learning model. In some non-limiting embodiments or aspects, when generating a final output using the updated critic machine learning model, ML optimization system 102 may generate a final output of an MLP machine learning model using the updated critic machine learning model, wherein the final output comprises a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM.

[0100] In some non-limiting embodiments or aspects, ML optimization system 102 may provide the final output as an input to the LLM, generate a result of the LLM based on the input, and perform an action based on the result of the LLM.

[0101] Referring now to FIGS. 3A-3E, FIGS. 3A-3E show schematic diagrams of implementation 300 of a process (e.g., process 200) for reducing order bias of machine learning models via feature re-ordering, according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, one or more of the steps of the process may be performed (e.g., completely, partially, etc.) by ML optimization system 102 (e.g., one or more devices of ML optimization system 102). In some non-limiting embodiments or aspects, one or more of the steps of the process may be performed (e.g., completely, partially, etc.) by another device or a group of devices separate from or including ML optimization system 102 (e.g., one or more devices of ML optimization system 102), ML model management database 104, and / or user device 106. In some non-limiting embodiments or aspects, as shown in FIGS.67P3321.DOCX Page 26 of 44Attorney Docket No. 08223-2503018 (9604W001)3A-3E, ML optimization system 102 may align a dataset, D = (x, y) I x eX, y X to the order bias of an LLM, which may be represented by f(«).

[0102] As shown by reference number 302 in FIG. 3A, ML optimization system 102 may receive a dataset (e.g., a training dataset) from user device 106. For example, ML optimization system 102 may receive the dataset that includes a plurality of data instances each having a plurality of features. In some non-limiting embodiments or aspects, each data instance x e X may have M features, where x = [xi, X2, ... , XM]. Let T = [T[0], T[i], . . ., T[M-1]] denotes a specific ordering of the features of data instance x, representing a ranking of the features xi through XM. Each T[t] e {xi, X2, ... , XM} indicates a feature ranked at position t; and T[0:t] denotes a slice of the trajectory with the top-ranked t features. For example, T= [X2, X3, XI] represents an order of features two, three, and one, and T[o:2] = [X2, X3]. In the example shown in FIGS. 3A-3E, the features may include at least age, gender, and education level of an individual.

[0103] As shown by reference number 304 in FIG. 3B, ML optimization system 102 may generate a feature trajectory based on the plurality of features. For example, ML optimization system 102 may generate the feature trajectory based on the plurality of features of the dataset using a critic machine learning model (e.g., a metacontroller).

[0104] In some non-limiting embodiments or aspects, the critic machine learning model may be represented as g(- | 9) : X x T - * R, where the critic machine learning model may be to estimate the value of ranking each feature at each location of a ranking. In this example, for 0 < t < M, with a slice of trajectory T[0:t, as input, a value of g(x, [T[0:t 1];j])eR, represents the value of trajectory [T[0:t 1];j], which also indicates the value of ranking feature j at position t, given the ranking of first t - 1 features T[0:t 1With this, a higher value of g(x, ) may indicate a better ranking of data features aligned with a preference of an LLM. Accordingly, ML optimization system 102 may recursively generate a trajectory of a plurality of features (e.g., M features) based on the following equation:67P3321.DOCX Page 27 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0106] A value function, V(T) , may be used to indicate whether an LLM can understand the data instance following a feature order in trajectory T. In some nonlimiting embodiments or aspects, a better understanding of the data instance may lead an LLM to align a data instance with a classification label. Therefore, V(T) may be defined based on the classification loss as follows:

[0108] Lf may denote the cross-entropy and V(T) may be opposite to the crossentropy loss such that an optimal trajectory T* can minimize the classification error.Further, V(T) defines the value of a complete trajectory, whereas for a slice of trajectory, T[0:t], the value function may be defined as v(T[0:t]), and the value function is defined as the maximal value of V( ~T ), such that ~T[0:t] = T[0:t], and the following equations follow:

[0110] where 0< / <1 denotes a discounting factor. The discounting factor may regulate how features ranked at different locations are influenced by cross entropy.

[0111] As shown by reference number 306 in FIG. 3C, ML optimization system 102 may estimate a value of loss of an output of an LLM based on the critic machine learning model. For example, ML optimization system 102 may estimate a value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model to provide an estimated value of loss of the output of the LLM. In some non-limiting embodiments or aspects, the output of the critic machine learning model may include a prediction of a value of loss of an output of the LLM based on an input, wherein the input comprises the feature trajectory of the plurality of features.67P3321.DOCX Page 28 of 44Attorney Docket No. 08223-2503018 (9604W001)

[0112] In some non-limiting embodiments or aspects, ML optimization system 102 may estimate the value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM. The final output of the LLM may be based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of all of the features of the plurality of features.

[0113] The value of loss of the output of the LLM may be calculated using the equation for V(T) for each ordered set of values, T, of all of the features of the plurality of features:

[0114]

[0115] As further shown in FIG. 3C, ML optimization system 102 may estimate the value of loss of the output of the LLM based on a plurality of slices of feature trajectory of the plurality of features. In some non-limiting embodiments or aspects, ML optimization system 102 may estimate the value of loss of the output of the LLM based on a plurality of slices of feature trajectory (e.g., where a slice is not a complete ordered set of features, but instead an ordered set of features that does not include all of the plurality of features) of the plurality of features in combination with the feature trajectory of the plurality of features as an ordered set.

[0116] ML optimization system 102 may estimate an MSE value of loss of an output of the LLM based on the feature trajectory of the plurality of features (e.g., each ordered set of the plurality of features) and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

[0117] As shown by reference number 308 in FIG. 3D, ML optimization system 102 may update one or more parameters of the critic machine learning model. For example, ML optimization system 102 may update one or more parameters (e.g., one or more weights and / or or one or more biases) of the critic machine learning model based on the value (e.g., the estimated value) of loss of the output of the LLM with regard to the feature trajectory of the plurality of features to provide an updated circuit machine learning model. In some non-limiting embodiments or aspects, ML optimization system 102 may update one or more parameters of the critic machine67P3321.DOCX Page 29 of 44Attorney Docket No. 08223-2503018 (9604W001) learning model to minimize the value of loss of the output of the LLM with regard to the feature trajectory of the plurality of features based on the equation:

[0119] In some non-limiting embodiments or aspects, the value function given by v(T[0:t]) = Yv^T[o:t+i]) has an iterative property with an initial value given by V(T) = rT.With this, ML optimization system 102 may update the parameters of g(xi;T[0:t]| 0) to minimize the MSE aligned with the value function v(T[0:t]) as follows:

[0120]

[0121] where v(T[0:t]) can be estimated based on its iterative property as follows:

[0123] As shown by reference number 310 in FIG. 3E, ML optimization system 102 may generate a final output using an updated critic machine learning model. For example, ML optimization system 102 may generate a final output of another machine learning model using the updated critic machine learning model. In some non-limiting embodiments or aspects, ML optimization system 102 may generate a final output of an MLP machine learning model using the updated critic machine learning model. In some non-limiting embodiments or aspects, the final output may include a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM.

[0124] In some non-limiting embodiments or aspects, ML optimization system 102 may provide the final output as an input to the LLM, generate a result of the LLM based on the input, and perform an action based on the result of the LLM. In some nonlimiting embodiments or aspects, ML optimization system 102 may perform an action based on the result of the LLM. In some non-limiting embodiments or aspects, ML optimization system 102 may perform a procedure associated protection of an account of a user (e.g., a user associated with user device 106) based on the result of the LLM. For example, if the result of the LLM indicates that the procedure is necessary, ML optimization system 102 may perform the procedure associated protection of the67P3321.DOCX Page 30 of 44Attorney Docket No. 08223-2503018 (9604W001) account of the user. In such an example, if the result of the LLM indicates that the procedure is not necessary, model management system 102 may forego performing the procedure associated protection of the account of the user. In some non-limiting embodiments or aspects, ML optimization system 102 may execute a fraud protection procedure based on the result of the LLM.

[0125] In some non-limiting embodiments or aspects, ML optimization system 102 may perform the action by providing the result of the LLM as a response to a request for inference for the LLM. In some non-limiting embodiments or aspects, ML optimization system 102 may perform the action by generating and transmitting an alert (e.g., an alert message) based on the result of the LLM. For example, ML optimization system 102 may perform the action by generating and transmitting an alert to user device 106 (e.g., a user associated with user device 106, such as a subject matter expert).

[0126] Referring now to FIG. 4, shown is a diagram of a non-limiting embodiment or aspect of exemplary environment 400 in which systems, methods, and / or products, as described herein, may be implemented. As shown in FIG. 4, environment 400 may include transaction service provider system 402, issuer system 404, customer device 406, merchant system 408, acquirer system 410, and communication network 412. In some non-limiting embodiments or aspects, each of ML optimization system 102, ML model database 104, and / or user device 106 of FIG. 1 may be implemented by (e.g., part of) transaction service provider system 402. In some non-limiting embodiments or aspects, at least one of ML optimization system 102, ML model database 104, and / or user device 106 of FIG. 1 may be implemented by (e.g., part of) another system, another device, another group of systems, or another group of devices, separate from or including transaction service provider system 402, such as issuer system 404, customer device 406, merchant system 408, acquirer system 410, and / or the like.

[0127] Transaction service provider system 402 may include one or more devices capable of receiving information from and / or communicating information to issuer system 404, customer device 406, merchant system 408, and / or acquirer system 410 via communication network 412. For example, transaction service provider system 402 may include a computing device, such as a server (e.g., a transaction processing server), a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, transaction service provider system 402 may be associated with a transaction service provider, as described herein. In some non-limiting67P3321.DOCX Page 31 of 44Attorney Docket No. 08223-2503018 (9604W001) embodiments or aspects, transaction service provider system 402 may be in communication with a data storage device, which may be local or remote to transaction service provider system 402. In some non-limiting embodiments or aspects, transaction service provider system 402 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device.

[0128] Issuer system 404 may include one or more devices capable of receiving information and / or communicating information to transaction service provider system 402, customer device 406, merchant system 408, and / or acquirer system 410 via communication network 412. For example, issuer system 404 may include a computing device, such as a server, a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, issuer system 404 may be associated with an issuer institution, as described herein. For example, issuer system 404 may be associated with an issuer institution that issued a credit account, debit account, credit card, debit card, and / or the like to a user associated with customer device 406.

[0129] Customer device 406 may include one or more devices capable of receiving information from and / or communicating information to transaction service provider system 402, issuer system 404, merchant system 408, and / or acquirer system 410 via communication network 412. Additionally or alternatively, each customer device 406 may include a device capable of receiving information from and / or communicating information to other customer devices 406 via communication network 412, another network (e.g., an ad hoc network, a local network, a private network, a virtual private network, and / or the like), and / or any other suitable communication technique. For example, customer device 406 may include a client device and / or the like. In some non-limiting embodiments or aspects, customer device 406 may or may not be capable of receiving information (e.g., from merchant system 408 or from another customer device 406) via a short-range wireless communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, and / or the like), and / or communicating information (e.g., to merchant system 408) via a short-range wireless communication connection.

[0130] Merchant system 408 may include one or more devices capable of receiving information from and / or communicating information to transaction service provider system 402, issuer system 404, customer device 406, and / or acquirer system67P3321.DOCX Page 32 of 44Attorney Docket No. 08223-2503018 (9604W001)410 via communication network 412. Merchant system 408 may also include a device capable of receiving information from customer device 406 via communication network 412, a communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, and / or the like) with customer device 406, and / or the like, and / or communicating information to customer device 406 via communication network 412, the communication connection, and / or the like. In some non-limiting embodiments or aspects, merchant system 408 may include a computing device, such as a server, a group of servers, a client device, a group of client devices, and / or other like devices. In some non-limiting embodiments or aspects, merchant system 408 may be associated with a merchant, as described herein. In some non-limiting embodiments or aspects, merchant system 408 may include one or more client devices. For example, merchant system 408 may include a client device that allows a merchant to communicate information to transaction service provider system 402. In some non-limiting embodiments or aspects, merchant system 408 may include one or more devices, such as computers, computer systems, and / or peripheral devices capable of being used by a merchant to conduct a transaction with a user. For example, merchant system 408 may include a POS device and / or a POS system.

[0131] Acquirer system 410 may include one or more devices capable of receiving information from and / or communicating information to transaction service provider system 402, issuer system 404, customer device 406, and / or merchant system 408 via communication network 412. For example, acquirer system 410 may include a computing device, a server, a group of servers, and / or the like. In some non-limiting embodiments or aspects, acquirer system 410 may be associated with an acquirer, as described herein.

[0132] Communication network 412 may include one or more wired and / or wireless networks. For example, communication network 412 may include a cellular network (e.g., a long-term evolution (LTE) network, a third generation (4G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, and / or the like), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber optic-based67P3321.DOCX Page 33 of 44Attorney Docket No. 08223-2503018 (9604W001) network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0133] The number and arrangement of systems, devices, and / or networks shown in FIG. 4 are provided as an example. There may be additional systems, devices, and / or networks; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or differently arranged systems, devices, and / or networks than those shown in FIG. 4. Furthermore, two or more systems or devices shown in FIG. 4 may be implemented within a single system or device, or a single system or device shown in FIG. 4 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of environment 400 may perform one or more functions described as being performed by another set of systems or another set of devices of environment 400.

[0134] Referring now to FIG. 5, shown is a diagram of example components of device 500, according to non-limiting embodiments or aspects. Device 500 may correspond to at least one of ML optimization system 102, ML model database 104, and / or user device 106 in FIG. 1 and / or at least one of transaction service provider system 402, issuer system 404, customer device 406, merchant system 408, and / or acquirer system 410 in FIG. 4, as an example. In some non-limiting embodiments or aspects, such systems or devices in FIG. 1 or FIG. 4 may include at least one device 500 and / or at least one component of device 500. The number and arrangement of components shown in FIG. 5 are provided as an example. In some non-limiting embodiments or aspects, device 500 may include additional components, fewer components, different components, or differently arranged components than those shown in FIG. 5. Additionally or alternatively, a set of components (e.g., one or more components) of device 500 may perform one or more functions described as being performed by another set of components of device 500.

[0135] As shown in FIG. 5, device 500 may include bus 502, processor 504, memory 506, storage component 508, input component 510, output component 512, and communication interface 514. Bus 502 may include a component that permits communication among the components of device 500. In some non-limiting embodiments or aspects, processor 504 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 504 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU),67P3321.DOCX Page 34 of 44Attorney Docket No. 08223-2503018 (9604W001) an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 506 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 504.

[0136] With continued reference to FIG. 5, storage component 508 may store information and / or software related to the operation and use of device 500. For example, storage component 508 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid-state disk, etc.) and / or another type of computer-readable medium. Input component 510 may include a component that permits device 500 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, input component 510 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 512 may include a component that provides output information from device 500 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 514 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 500 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 514 may permit device 500 to receive information from another device and / or provide information to another device. For example, communication interface 514 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0137] Device 500 may perform one or more processes described herein. Device 500 may perform these processes based on processor 504 executing software instructions stored by a computer-readable medium, such as memory 506 and / or storage component 508. A computer-readable medium may include any non- transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical67P3321.DOCX Page 35 of 44Attorney Docket No. 08223-2503018 (9604W001) storage devices. Software instructions may be read into memory 506 and / or storage component 508 from another computer-readable medium or from another device via communication interface 514. When executed, software instructions stored in memory 506 and / or storage component 508 may cause processor 504 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software. The term “configured to,” as used herein, may refer to an arrangement of software, device(s), and / or hardware for performing and / or enabling one or more functions (e.g., actions, processes, steps of a process, and / or the like). For example, “a processor configured to” may refer to a processor that executes software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0138] Although embodiments have been described in detail for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect.67P3321.DOCX Page 36 of 44

Claims

Attorney Docket No. 08223-2503018 (9604W001)WHAT IS CLAIMED IS:1 . A system, comprising: at least one processor configured to: receive a training dataset comprising a plurality of data instances for a plurality of features; generate a feature trajectory based on the plurality of features, wherein the feature trajectory comprises a predicted ordered set of values of at least a subset of features of the plurality of features; estimate a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model comprises a prediction of a value of loss of an output of the LLM based on an input, and wherein the input comprises the feature trajectory; update one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generate a final output using the updated critic machine learning model, wherein the final output comprises a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

2. The system of claim 1 , wherein, when receiving the training dataset comprising the plurality of data instances for the plurality of features, the at least one processor is configured to: receive an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.

3. The system of claim 1 , wherein, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate the value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM,67P3321.DOCX Page 37 of 44Attorney Docket No. 08223-2503018 (9604W001) wherein the final output of the LLM is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of all of the features of the plurality of features.

4. The system of claim 1 , wherein, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate the value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of a subset of features of the plurality of features.

5. The system of claim 1 , wherein, when updating the one or more parameters of the critic machine learning model, the at least one processor is configured to: update one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

6. The system of claim 1 , wherein, when estimating a value of loss of an output of the LLM, the at least one processor is configured to: estimate a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

7. The system of claim 1 , wherein, when generating a final output using the updated critic machine learning model, the at least one processor is configured to: generate a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final67P3321.DOCX Page 38 of 44Attorney Docket No. 08223-2503018 (9604W001) output comprises a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the at least one processor is further configured to: provide the final output as an input to the LLM; generate a result of the LLM based on the input; and perform an action based on the result of the LLM.

8. A computer-implemented method, comprising: receiving, with at least one processor, a training dataset comprising a plurality of data instances for a plurality of features; generating, with at least one processor, a feature trajectory based on the plurality of features, wherein the feature trajectory comprises a predicted ordered set of values of at least a subset of features of the plurality of features; estimating, with at least one processor, a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model comprises a prediction of a value of loss of an output of the LLM based on an input, and wherein the input comprises the feature trajectory; updating, with at least one processor, one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generating, with at least one processor, a final output using the updated critic machine learning model, wherein the final output comprises a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

9. The computer-implemented method of claim 8, wherein receiving the training dataset comprising the plurality of data instances for the plurality of features comprises: receiving an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.67P3321.DOCX Page 39 of 44Attorney Docket No. 08223-2503018 (9604W001)10. The computer-implemented method of claim 8, wherein estimating a value of loss of an output of the LLM comprises: estimating the value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, wherein the final output of the LLM is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of all of the features of the plurality of features.1 1 . The computer-implemented method of claim 8, wherein estimating a value of loss of an output of the LLM comprises: estimating the value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of a subset of features of the plurality of features.

12. The computer-implemented method of claim 8, wherein updating the one or more parameters of the critic machine learning model comprises: updating one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.

13. The computer-implemented method of claim 8, wherein estimating a value of loss of an output of the LLM comprises: estimating a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.

14. The computer-implemented method of claim 8, wherein generating a final output using the updated critic machine learning model comprises:67P3321.DOCX Page 40 of 44Attorney Docket No. 08223-2503018 (9604W001) generating a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output comprises a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the method further comprises: providing the final output as an input to the LLM; generating a result of the LLM based on the input; and performing an action based on the result of the LLM.

15. A computer program product, comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, causes the at least one processor to: at least one processor configured to: receive a training dataset comprising a plurality of data instances for a plurality of features; generate a feature trajectory based on the plurality of features, wherein the feature trajectory comprises a predicted ordered set of values of at least a subset of features of the plurality of features; estimate a value of loss of an output of a large language model (LLM) based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated value of loss of the output of the LLM, wherein the output of the critic machine learning model comprises a prediction of a value of loss of an output of the LLM based on an input, and wherein the input comprises the feature trajectory; update one or more parameters of the critic machine learning model based on the estimated value of loss of the output of the LLM to provide an updated critic machine learning model; and generate a final output using the updated critic machine learning model, wherein the final output comprises a final ordered set of features that provides maximized accuracy of a prediction of the LLM.

16. The computer program product of claim 15, wherein, the program instructions that cause the at least one processor to receive the training dataset67P3321.DOCX Page 41 of 44Attorney Docket No. 08223-2503018 (9604W001) comprising the plurality of data instances for the plurality of features, cause the at least one processor to: receive an ordered set of values of the plurality of features for each data instance, wherein the ordered set of values is based on a number of features included in the training dataset.

17. The computer program product of claim 15, wherein, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate the value of loss of the output of the LLM based on the feature trajectory of the plurality of features, an output of a critic machine learning model, and a final output of the LLM to provide an estimated value of loss of the output of the LLM, wherein the final output of the LLM is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of all of the features of the plurality of features.

18. The computer program product of claim 15, wherein, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate the value of loss of the output of the LLM based on the feature trajectory of the plurality of features and an output of the critic machine learning model, wherein the output of the critic machine learning model is based on an input, wherein the input comprises the feature trajectory of the plurality of features, and wherein the feature trajectory of the plurality of features comprises an ordered set of values of a subset of features of the plurality of features.

19. The computer program product of claim 15, wherein, the program instructions that cause the at least one processor to update the one or more parameters of the critic machine learning model, cause the at least one processor to: update one or more parameters of the critic machine learning model to minimize the estimated value of loss of the output of the LLM to provide the updated critic machine learning model.67P3321.DOCX Page 42 of 44Attorney Docket No. 08223-2503018 (9604W001)20. The computer program product of claim 15, wherein, the program instructions that cause the at least one processor to estimate a value of loss of an output of the LLM, cause the at least one processor to: estimate a mean squared error (MSE) value of loss of an output of the LLM based on the feature trajectory of the plurality of features and an output of a critic machine learning model to provide an estimated MSE value of loss of the output of the LLM.21 . The computer program product of claim 15, wherein, the program instructions that cause the at least one processor to generate a final output using the updated critic machine learning model, cause the at least one processor to: generate a final output of a multilayer perceptron (MLP) machine learning model using the updated critic machine learning model, wherein the final output comprises a final ordered set of features to be provided as a prompt to the LLM that provides maximized accuracy of a prediction of the LLM; and wherein the program instructions further cause the at least one processor to: provide the final output as an input to the LLM; generate a result of the LLM based on the input; and perform an action based on the result of the LLM.67P3321.DOCX Page 43 of 44

Images