Human consistency assessment system and method for automatically building optimization model using large language model

Abstract

The invention provides a method for evaluating the capability of generating an accurate hypothesis model (HM) of a linear programming application problem (LPWP) for a large language model (LLM), and provides a method for evaluating the capability of generating an accurate hypothesis model (HM) of a linear programming application problem (LPWP) for the LLM. The attribute bipartite graph of the HM is transformed into an attribute bipartite graph of a ground truth value model of the LPWP. The number of transform steps required for the transform effectively measures the ability of the LLM to generate an accurate HM. An attribute bipartite graph is used to provide order invariance for LPWP variables and constraints. The invention further provides a corresponding device and system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the U.S. patent application filed with the U.S. Patent Office on October 18, 2023, with application number US18381239 and application name “SYSTEMS AND METHODS FOR TOWARDS HUMAN-ALIGNED EVALUATION FOR AUTO-FORMULATING OPTIMIZATION MODELING WITH LARGE LANGUAGE MODELS”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present invention relates to the fields of machine learning and natural language processing, and more particularly to a system and method for obtaining a mathematical form of an optimization problem using a large language model (LLM). Background Art

[0004] Evaluating predictive models, especially those generated by machine learning models such as LLMs for optimization problems such as linear programming word problems (LPWPs), presents a unique set of challenges. One such challenge revolves around the concept of permutation invariance, which is difficult for existing evaluation methods to provide strong support for. Ensuring that the model's predictions remain unchanged when the input variables are permuted is critical for reliable evaluation. Additionally, confidently determining model equivalence is another challenge. Some evaluation methods may incorrectly consider a predictive model to be equivalent to its reference model simply because the model produces an optimal objective value that is consistent with the reference model. This error can occur even if the predictive model only ignores a single constraint. In some cases, seemingly dissimilar models may produce "infeasible" results, and the optimal objective value is incorrectly considered to be the correct prediction due to these complexities. Addressing these challenges that arise when evaluating predictive models for optimization problems is important to increasing the reliability and applicability of the models in real-world scenarios.

[0005] Therefore, it is desirable to make improvements in obtaining the mathematical form of optimization problems using LLM.

[0006] The purpose of this background information is to disclose information believed by the applicant to be potentially relevant to the present invention. It is not necessary to admit, nor should it be construed, that any of the above information constitutes prior art against the present invention. Summary of the invention

[0007] A human consistent evaluation device, system and method for automatically constructing an optimization model using an LLM. According to one aspect, the present invention provides a computer-implemented method for evaluating the accuracy of a hypothesis model (HM) relative to a ground truth model (GTM). The method includes obtaining a graph of the HM. The graph of the HM includes HM constraint vertices associated with HM constraints, HM variable vertices associated with HM variables, and HM edges connecting the HM constraint vertices to the HM variable vertices. The method also includes obtaining a graph of the GTM. The graph of the GTM includes GTM constraint vertices associated with GTM constraints, GTM variable vertices associated with GTM variables, and GTM edges connecting the GTM constraint vertices to the GTM variable vertices. In addition, the method includes: transforming the graph of the HM into the graph of the GTM through a series of transformation steps. The total number of transformation steps used to transform the graph of the HM into the graph of the GTM is a measure of the accuracy of the HM.

[0008] In some embodiments of the first aspect, transforming the graph of the HM into the graph of the GTM comprises: automatically transforming the graph of the HM into the graph of the GTM using a graph edit distance calculation algorithm. In some embodiments, the graph edit distance calculation algorithm is based on an A* algorithm. In some embodiments, the graph edit distance calculation algorithm is a depth-first graph edit distance algorithm.

[0009] In some embodiments of the first aspect, the method may further include: obtaining the HM by inputting a mathematical word problem into a LLM to generate the HM. Inputting the mathematical word problem into the LLM may include: inputting a mathematical application optimization problem into the LLM. In some embodiments, generating the HM may include: obtaining the HM variables from the LLM; a lower bound of each corresponding HM variable in the HM variables; an upper bound of each corresponding HM variable in the HM variables. Generating the HM may further include obtaining at least one HM function that depends on the HM variables. The at least one function may define an output to be optimized according to the mathematical word problem. Generating the HM may further include obtaining one or more HM constraint expressions. The one or more HM constraint expressions may define the HM constraints imposed on the HM variables. Generating the HM may further include: transforming the at least one HM function and the one or more HM constraint expressions into a general linear programming model of the mathematical word problem to obtain the HM.

[0010] In some embodiments of the first aspect, the GTM may be associated with a lower bound and an upper bound for each corresponding one of the GTM variables. The GTM may also have at least one GTM function that depends on the GTM variables, wherein the at least one GTM function defines an output to be optimized according to the mathematical word problem. In addition, the GTM may also have one or more GTM constraint expressions, wherein the one or more GTM constraint expressions define the GTM constraints imposed on the GTM variables.

[0011] In some embodiments of the first aspect, obtaining the graph of the HM may include forming an HM attribute bipartite graph by generating the HM constraint vertices according to the HM variables and the one or more HM constraint expressions. Forming the HM attribute bipartite graph may also include generating the HM variable vertices according to the following: the HM variables; the lower bounds and the upper bounds of each corresponding HM variable in the HM variables; the at least one HM function. Generating the HM attribute bipartite graph may also include generating the HM edges according to the one or more HM constraint expressions, wherein the HM edges connect the HM constraint vertices to the HM variable vertices. The HM constraint vertices may form a first set of HM vertices. The HM variable vertices may form a second set of HM vertices. The first set of HM vertices and the second set of HM vertices may be disjoint. Obtaining the graph of the GTM may include forming a GTM attribute bipartite graph by generating the GTM constraint vertices according to the GTM variables and the one or more GTM constraint expressions. Forming the GTM attribute bipartite graph may also include generating the GTM variable vertices based on: the GTM variables; the lower bounds and the upper bounds of each corresponding GTM variable in the GTM variables; the at least one GTM function. Forming the GTM attribute bipartite graph may also include generating the GTM edges based on the one or more GTM constraint expressions, wherein the GTM edges connect the GTM constraint vertices to the GTM variable vertices. The GTM constraint vertices may form a first GTM vertex set. The GTM variable vertices may form a second GTM vertex set. The first GTM vertex set and the second GTM vertex set may be disjoint.

[0012] In some embodiments of the first aspect, the HM constraint vertex is associated with an HM constraint value, the HM variable vertex is associated with an HM variable value, and the series of transformation steps may include at least one of the following: replacing the HM constraint value with a different HM constraint value; replacing the HM variable value with a different HM variable value; adding an HM constraint vertex; adding an HM variable vertex; deleting the HM constraint vertex; deleting the HM variable vertex; adding an edge connecting the HM constraint vertex to the HM variable vertex; deleting a second edge connecting the second HM constraint vertex to the second HM variable vertex; replacing the weight of any edge with a different weight.

[0013] In some embodiments of the first aspect, the method may further include obtaining an accuracy score based on the total number of the transformation steps. The accuracy score may indicate the ability of the LLM to accurately generate the HM. The method may further include: modifying the LLM based on the accuracy score; iteratively performing actions (A) to (G) until a stopping criterion is met, wherein actions (A) to (G) are described below. (A) generating other HMs by inputting other math word problems into the LLM, wherein the other math word problems are associated with other GTMs; (B) obtaining graphs of the other HMs; (C) obtaining graphs of the other GTMs; (D) transforming the graphs of the other HMs into the graphs of the other GTMs through a series of other transformation steps, wherein the total number of the other transformation steps for transforming the graphs of the other HMs into the graphs of the other GTMs is a measure of the accuracy of the other HMs; (E) obtaining other accuracy scores based on the total number of the other transformation steps, wherein the other accuracy scores indicate the ability of the LLM to accurately generate the other HMs; (F) modifying the LLM based on the accuracy score; and (G) determining whether the stopping criteria are met.

[0014] In some embodiments of the first aspect, obtaining a first graph representation may include obtaining a first graph representation set of a first optimization model set generated by a machine learning model based on a first optimization problem set. Each graph representation in the first graph representation set may correspond to an optimization model in the first optimization model set based on a corresponding optimization problem in the first optimization problem set. Obtaining a second graph representation may include obtaining a second graph representation set of a second optimization model set, wherein the second optimization model set is used as a ground truth of the first optimization problem set. Each graph representation in the second graph representation set may correspond to an optimization model in the second optimization model set based on a corresponding optimization problem in the first optimization problem set. Obtaining a minimum number of editing operations may include obtaining a set of editing operations for transforming the first graph representation set into the second graph representation set.

[0015] In some embodiments of the first aspect, obtaining a set of editing operations for transforming the first graph representation set into the second graph representation set may include: for each optimization problem, obtaining a series of one or more editing operations in the set of editing operations for transforming a corresponding graph in the first graph representation set into a corresponding graph representation in the second graph representation set.

[0016] In some embodiments of the first aspect, the method may further include obtaining a set of graph edit distances (GEDs) corresponding to the set of edit operations. Each GED may correspond to each of the optimization problems and indicate a metric of the corresponding one or more sequences of edit operations. The method may further include obtaining one or more of the following:

[0017] Based on an exact match rate of the GED set, wherein the exact match rate may indicate a proportion of the first optimization model set having corresponding GEDs in the GED set indicating an equivalent match; based on an average value of the GEDs of the GED set.

[0018] In some embodiments of the first aspect, the method may further include adjusting one or more weights of the machine learning model based on the set of edit operations. The machine learning model may include a plurality of nodes connected to each other via a plurality of connections. The machine learning model also includes the one or more weights corresponding to the plurality of connections.

[0019] In some embodiments of the first aspect, the method may further include generating a training data set based on the set of editing operations, wherein the training data set includes a second set of optimization problems. The method may further include: training the machine learning model using the second training data set set.

[0020] In some embodiments of the first aspect, training the machine learning model using the training data set may include: inputting the second training data set into the machine learning model; obtaining a third set of optimization models generated by the machine learning model based on the second set of optimization problems. Training the machine learning model using the training data set may also include obtaining a third set of graph representations of the third set of optimization models, wherein each graph representation in the third set of graph representations corresponds to an optimization model in the third set of optimization models based on a corresponding optimization problem in the second set of optimization problems. Training the machine learning model using the training data set may also include obtaining a fourth set of graph representations of a fourth set of optimization models, wherein the fourth set of optimization models is used as the ground truth of the second set of optimization problems, and each graph representation in the fourth set of graph representations corresponds to an optimization model in the fourth set of optimization models based on the corresponding optimization problem in the second set of optimization problems. In addition, training the machine learning model using the training data set may also include obtaining a second set of editing operations for transforming the third set of graph representations into the fourth set of graph representations, and adjusting the one or more weights of the machine learning model based on the second set of editing operations.

[0021] In some embodiments of the second aspect, the method may further include obtaining a set of reward signals based on the set of edit operations, wherein each reward signal indicates a quality of an optimization model in the first set of optimization models relative to a corresponding optimization model in the second set of optimization models. Adjusting one or more weights of the machine learning model based on the set of edit operations may include: adjusting the one or more weights of the machine learning based on the set of reward signals using reinforcement learning.

[0022] In a third aspect, the present invention provides a computer-implemented method for evaluating the accuracy of an LLM, wherein the accuracy is to generate an HM corresponding to a LPWP. The method includes obtaining a plurality of HMs corresponding to a plurality of LPWPs from the LLM, wherein each LPWP in the plurality of LPWPs is associated with a corresponding GTM, all the corresponding GTMs form a plurality of GTMs, and each HM in the plurality of HMs is associated with a corresponding GTM in the plurality of GTMs. The method further includes: for each HM in the plurality of HMs: obtaining a graph of the HM; obtaining a graph of the GTM associated with the HM. The graph of the HM includes HM constraint vertices associated with HM constraints, HM variable vertices associated with HM variables, and HM edges connecting the HM constraint vertices to the HM variable vertices. The graph of the GTM includes GTM constraint vertices associated with GTM constraints, GTM variable vertices associated with GTM variables, and GTM edges connecting the GTM constraint vertices to the GTM variable vertices. The method further comprises transforming the graph of the HM into the graph of the GTM through a series of transformation steps having a total number of transformation steps. The method further comprises calculating a score representing the accuracy of the LLM, wherein the score is a function of the total number of all the transformation steps used to transform all the HM graphs associated with the plurality of HMs into all corresponding GTM graphs associated with the plurality of GTMs.

[0023] In a fourth aspect, the present invention provides a method, the method comprising obtaining a first graph representation of a hypothetical optimization problem model generated by a machine learning model based on an optimization problem. The method further comprises obtaining a second graph representation of a reference optimization problem model based on the optimization problem. The method further comprises obtaining a minimum number of edit operations for transforming the first graph representation into the second graph representation. The minimum number of edit operations indicates the performance of the machine learning model.

[0024] In some embodiments of the fourth aspect, the machine learning model may be an LLM, and the optimization problem may be one of the following: an LPWP; a mixed integer linear programming problem; a quadratic programming problem; a quadratic constrained quadratic programming problem. The first graph representation may be a first attribute bipartite graph. The second graph representation may be a second attribute bipartite graph. The minimum number of edit operations may be a function of a graph edit distance between the first attribute bipartite graph and the second attribute bipartite graph.

[0025] In some embodiments of the fourth aspect, the optimization problem may include a set of constraints and a set of decision variables, and the method may further include: obtaining a first linear programming general form corresponding to the hypothetical optimization problem model; obtaining a second linear programming general form corresponding to the reference optimization problem model.

[0026] In some embodiments of the fourth aspect, the minimum number of edit operations may be a minimum sequence of edit operations for transforming the first attributed bipartite graph into the second attributed bipartite graph; in some embodiments, the minimum sequence of edit operations involves one or more of the following: vertices in the first vertex set, vertices in the second vertex set, edges in the edge set. In some embodiments, the minimum sequence of edit operations may involve one or more of the following: an insertion operation, a deletion operation, and a replacement operation.

[0027] In some embodiments of the fourth aspect, the method may further include: obtaining a total cost of the minimum editing operation sequence based on a cost value assigned to each editing operation in the minimum editing operation sequence. The total cost may indicate the performance of the machine learning model.

[0028] According to another aspect, an apparatus may be provided, comprising modules or electronic components for executing one or more of the methods and systems described herein.

[0029] According to one aspect, a device may be provided, wherein the device comprises: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute one or more of the methods and systems described in this article.

[0030] According to another aspect, a computer readable medium may be provided, wherein the computer readable medium stores program code executed by a device, the program code being used to perform one or more of the methods and systems described herein.

[0031] According to one aspect, a chip may be provided, wherein the chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute one or more of the methods and systems described herein. Aspects may also include the memory.

[0032] Other aspects of the present invention provide apparatus and systems for implementing the method according to the first aspect disclosed herein. For example, a wireless station and an access point may be configured with a machine-readable memory containing instructions, which, when executed by a processor of these devices, configures the device to perform one or more of the methods and systems described herein.

[0033] Embodiments are described above in conjunction with various aspects of the present invention, and these embodiments can be implemented based on these aspects. It will be appreciated by those skilled in the art that embodiments can be implemented in conjunction with the aspects describing these embodiments, but can also be implemented together with other embodiments of this aspect. When the embodiments are mutually exclusive or inconsistent with each other, this is obvious to those skilled in the art. Some embodiments can be described in conjunction with an aspect, but can also be applied to other aspects, which is obvious to those skilled in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0035] Figure 1 An example in which the optimization problem according to the present invention is LPWP is shown.

[0036] Figure 2 An example of the difficulties associated with specification accuracy is shown.

[0037] Figure 3 Two scenarios are shown where the execution accuracy is insufficient.

[0038] Figure 4 An attributed bipartite graph according to an embodiment of the present invention is shown.

[0039] Figure 5A An embodiment of a graph editing path between a hypothetical graph and a reference graph is shown.

[0040] Figure 5B An embodiment of another graph editing path between a hypothetical graph and a reference graph is shown.

[0041] Figure 6 A flow chart of an embodiment of a method according to the invention is shown.

[0042] Fig. 7A The flow of an embodiment of the method according to the present invention is shown.

[0043] Figure 7B A flow chart of an embodiment of a method according to the invention is shown.

[0044] Figure 8 The flow of an embodiment of the method according to the present invention is shown.

[0045] Fig. 9 An exemplary arrangement according to the invention is shown.

[0046] It should be noted that throughout the drawings, like features are identified by like reference numerals. DETAILED DESCRIPTION

[0047] Apparatus, systems and methods may be provided for human-consistent evaluation of optimization models automatically constructed using LLMs.

[0048] A math word problem as used in this article refers to a problem whose purpose is to provide an expression of a solution in response to a given math problem description (or linear programming math problem description) presented as a textual narrative rather than mathematical symbols. Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and humans through natural language, enabling computers to understand, interpret, and generate human language in a meaningful way. Text generation can refer to the generation of human-interpretable text, and the text generation task is a subfield of NLP that aims to create human-interpretable text. Entity recognition can refer to the task of identifying and classifying named entities such as people, organizations, locations, and other specific terms in text.

[0049] Arithmetic problems are math problems that use basic operations such as addition, subtraction, multiplication, and division to perform calculations. Algebraic problems involve solving for unknown values ​​using mathematical symbols, variables, and expressions that represent relationships between quantities.

[0050] An optimization problem is a mathematical problem whose goal is to find the best possible solution from a set of available choices, subject to certain constraints or restrictions. The goal is usually to maximize or minimize a specific objective function while ensuring that the solution satisfies the constraints.

[0051] LPWP is an optimization problem in which both the objective function and the constraints are linear. For example, LPWP can refer to a mathematical problem that involves optimizing (e.g., maximizing or minimizing) a linear objective function while satisfying a set of linear constraints. Linear programming can refer to a mathematical technique for finding the best result in a mathematical model with linear relationships. In the context of word problems, these scenarios often represent real-world situations where highly diverse human languages ​​describe real-world problems.

[0052] An LPWP can have an objective. The objective can define the purpose of the problem (e.g., minimize cost, maximize calcium, maximize number of participants, etc.). An LPWP can also have one or more constraints. These constraints are considerations of the problem and can be real-world limitations (e.g., the number of people is not negative, the cargo on the truck is less than 20kg due to weight restrictions) or user-specified wishes (e.g., at least 10 items are sold). An LPWP can also have one or more decision variables (variables) whose values ​​can be changed to achieve the purpose of the optimization problem. An optimal solution to an optimization problem can refer to one or more values ​​of one or more decision variables that optimize the objective function while satisfying the constraints.

[0053] The mathematical problem description can use only mathematical content or natural language, or use both natural language and mathematical content to describe the optimization problem. For example, the problem description may be how an expert describes the optimization problem to an operations research (OR) expert.

[0054] LLM can refer to a machine learning model trained on a large amount of text data, which is used to generate human-like text based on given input or instructions. The term "end-to-end" about a process can mean that the process covers all stages or components of the system, ensuring that the task is fully completed from beginning to end.

[0055] A false positive can refer to an error in data reporting where a test result incorrectly indicates that a condition exists, such as a diagnostic test erroneously showing a positive result for a disease that does not actually exist. A false negative can refer, for example, to a test result that incorrectly indicates that a condition (e.g., a disease or a positive match) does not exist, but the condition does exist.

[0056] A mathematical model form may refer to a method of representing a linear programming optimization problem. A HM may refer to a mathematical model form obtained by an LLM. A reference model (or GTM) may refer to a mathematical model form used as a ground truth in test data for comparison with the HM to evaluate the performance of the LLM.

[0057] Permutation invariance can refer to the property of a function or system that remains the same or invariant when the elements or items in a set are rearranged (permuted) in a different order. For example, if two mathematical model forms differ only in the order of variables and constraints, they are still technically equivalent. An evaluation metric with permutation invariance can correctly reveal that the two forms are equivalent.

[0058] Model equivalence (matching) can refer to the equivalence of two optimization models, that is, the two models contain the same information covered in the problem description.

[0059] A mathematical programming system (MPS) file is a widely supported format for linear and mixed integer programming solvers. Solvers (or optimization solvers) can refer to software tools that take model instances as input and implement standard algorithms to solve mathematical optimization problems. The term "parsing" can refer to the process of converting an input text string into smaller fragments, also called tokens. This process is performed by a parser.

[0060] Graph edit distance may refer to a metric that quantifies the similarity between two graphs by calculating the minimum cost of transforming one graph into another through a series of edit operations. For example, the edit operations may include one or more of the following: vertex insertion, vertex deletion, vertex replacement, edge insertion, edge deletion, and edge replacement, etc.

[0061] A bipartite graph (or attributed bipartite graph) is a graph representation whose vertices can be partitioned into two disjoint sets such that no two vertices in the same set are adjacent.

[0062] Math word problems (MWPs) are a fundamental yet challenging NLP task that has received considerable attention in recent years. Essentially, the goal of MWPs is to provide solution expressions in response to a given math problem description. Most previous studies on this task have focused on efficiently returning solutions to elementary arithmetic and algebraic problems, as these problems are easy to model and test, and computational methods for these problems can be evaluated in a relatively straightforward and standardized manner. However, another class of math word problems, namely LPWPs, remains largely underexplored.

[0063] Figure 1 An example of an optimization problem is shown as a LPWP, which is represented by a text problem description 102 and its paired optimization model form 104, wherein the problem description contains basic information of the model. The optimization model form can include one or more components: an objective 108, a constraint 110, and a decision variable (or variable 106).

[0064] While LPWP can more realistically reflect real-world decision-making processes, thus bringing great benefits to the field of operations research (OR), the reasoning-intensive nature of LPWP may require that previous neural solutions be decomposed into sub-steps (e.g., first entity recognition, then text generation), which inevitably leads to error accumulation. In addition, the data sparsity problem also brings additional difficulties to neural methods, making it difficult to consistently achieve reliable and robust performance. To address the above difficulties, the problem description can be formulated as an instruction that guides the LLM to generate an optimized model 104 as an answer in an easy-to-process format using an empirically adjusted prompt template. This modeling process can be end-to-end and does not require large-scale ground truth data.

[0065] Some existing evaluation methods to assess the effectiveness of LLM for LPWP include specification accuracy and implementation accuracy.

[0066] The canonical accuracy is based on the statement-level match between the hypothesis model and the reference model, where a statement is by definition a representation of the optimization objective or constraint. Specifically, the canonical accuracy of a LPWP problem can be calculated as follows:

[0067]

[0068] Among them, for a given problem i, D i is the number of actual claims in the ground truth model. i represents the number of statements in the prediction that do not match any actual statement, while the false negatives FN i It indicates the number of statements that are not in the forecast statement but are actually correct. i +FN i May exceed D i , so min is used to prevent the accuracy score from being negative.

[0069] Execution accuracy is similar to the commonly adopted code generation evaluation framework, which focuses on the functional correctness of program predictions. The execution evaluation strategy of LPWP aims to evaluate the correctness of the optimization model assumed by LLM by comparing the optimal solution between the assumed model and the reference model. The process includes: converting the mathematical model form into a mathematical programming system (MPS) file format, which can then be input into the solver to derive the optimal objective value. If the optimal objective value between the assumed model and the reference model matches exactly, the prediction can be considered successful.

[0070] Figure 2 An example of canonical accuracy is shown. Although canonical accuracy can be constrained by some permutation ( Figure 2 202), but the canonical accuracy is still limited by a strict assumption that the LLM must follow the same variable mapping order as the variable mapping in the ground truth model. For example, even if a.X+bY≤c and b.Y+aX≤c are basically equivalent, the canonical measure will declare them unequal due to the change in variable order. As shown in Example 210, the LLM prediction 214 simply swaps the variables and constraints in the ground truth 212. However, in this scenario, the canonical method exhibits significantly lower accuracy. Therefore, since LLM cannot always follow the same fictitious variable names, canonical accuracy can easily introduce over-penalization.

[0071] In example 202, the order of constraints is different between the ground truth 204 and the LLM prediction 206, but the order of variables remains the same. In this case, the canonical metric is still valid. In example 210, the order of two variables x and y in the LLM prediction 214 is swapped compared to the ground truth 212, and all claims are affected by this swap accordingly. In example 210, the canonical accuracy is likely to err, introducing more false negatives when matching claims.

[0072] Execution accuracy cannot determine with confidence that the HM (forecast model) is equivalent or an exact match to the GTM. Therefore, execution accuracy cannot be used as a perfect indicator of model equivalence. Figure 3 Two scenarios are shown where the execution accuracy makes an incorrect judgment, and two possible scenarios are shown where the execution evaluation method makes an incorrect judgment on whether the ground truth and LLM predictions are equivalent.

[0073] Scenario 302 shows an LLM prediction 306 that ignores a number of constraints compared to the ground truth 304, but produces an optimal objective value that is consistent with the reference model 304. In the second scenario 310, a pair of non-identical models 312 and 314 produce an "infeasible" result as the optimal objective value, which will also be considered a correct prediction at the execution accuracy.

[0074] In order to address the deficiencies of existing evaluation indicators, according to one aspect, an enhanced evaluation indicator can be provided. According to one aspect, a simple and effective evaluation indicator can be provided for evaluating the modeling performance of a machine learning model (e.g., LLM) on an optimization problem (e.g., LPWP). According to one aspect, the evaluation indicator can achieve variable or constraint permutation invariance and reliable model exact match identification.

[0075] The evaluation metric or evaluation strategy can be based on graph edit distance. According to one aspect, the evaluation metric can involve extracting a mathematical model form from the ground truth (reference model) and the answer (hypothesis or prediction model) provided by the LLM using a robust parser. The mathematical model form can then be converted into a bipartite graph with weighted edges that connect vertices representing variables and constraints. The graph edit distance between the paired hypothesis graph and the reference graph can be calculated. The graph edit distance can be used as a metric to evaluate the accuracy of the predictions made by the LLM. The enhanced evaluation metric can overcome the limitations of the existing evaluation metrics described herein.

[0076] The enhanced evaluation metrics may achieve permutation invariance. Compared with the canonical measurement method, the enhanced evaluation metrics can better adapt to the order differences of variables and constraints between the reference model and the HM, because the order changes of these constraints or variables do not inherently indicate that the two models are different.

[0077] The enhanced evaluation metric also enables exact match identification. Compared to executable measurements, the enhanced evaluation metric can detect equivalence or exact matches between HM and reference models with greater confidence. The enhanced evaluation metric also addresses the executable measurement problem where two models with the same (infeasible) optimal solution may not be equivalent.

[0078] According to one aspect, a method (or evaluation strategy) can be provided for evaluating the performance of an LLM based on an optimization problem. The method can be based on a graph edit distance between a reference model and an HM generated by the LLM corresponding to the reference model. The method can successfully address the defects in execution and specification accuracy as described herein. In addition, the method can achieve greater consistency with human perception because a smaller graph edit distance indicates fewer errors in the HM generated by the LLM and better modeling capabilities of the LLM.

[0079] According to one aspect, the method may include: converting the mathematical form of the HM generated by the LLM and the corresponding reference model into a general form of linear programming (LP). The method may also include: converting the general LP model form into a graph representation. In some embodiments, the graph representation may be based on an attributed bipartite graph. The attributed bipartite graph may include a first type of vertex that is a variable vertex and a second type of vertex that is a constraint vertex. Each variable vertex may have one or more attributes, including: an upper bound, a lower bound, and a coefficient relative to a target. Each constraint vertex may have one or more attributes, including an upper bound and a lower bound. Each edge between a variable vertex and a constraint vertex may represent a coefficient of the variable vertex relative to the constraint vertex.

[0080] Thus, the paired original hypothesis and reference model (GTM) may be transformed into a graph using any suitable rule-based graph transformer. The method may also include determining a graph edit distance between a graph representation of the HM and a graph representation of a corresponding reference model. Determining the graph edit distance may include applying a method for calculating the graph edit distance to measure a sequence of basic edit operations with the lowest cost to transform the graph representation of the HM into a graph representation of the corresponding reference model. The basic edit operations may include one or more of: inserting, deleting, and replacing one or more of vertices and edges. In some embodiments, a graph edit distance of 0 may be confidently interpreted as an exact match between the ground truth and the LLM prediction. In addition, since attributes are attached to constraints or variables, this method may also have permutation invariance.

[0081] According to one aspect, a method for automatically constructing an optimization model using LLM can be provided, which can improve accuracy and robustness evaluation. The method can address the shortcomings of existing methods through permutation invariance and better identification of model exact matches.

[0082] The method can also integrate available information in other forms (such as textual interpretations). Figure 1 , each model component (claim) is followed by a textual explanation generated by the LLM (e.g., “apple planting area, acres” and “pear planting area, acres” are both textual explanations generated by the LLM). The method can add this explanation as another attribute to the corresponding vertex in the graph representation. Furthermore, a cost function can be defined for this attribute, so that the evaluation metric may be more granular and further aligned with human perspectives. In this way, the method can be extended to achieve explanation integration.

[0083] According to one aspect, the method can implement error backtracking. In some embodiments, the method can obtain a score or metric indicating the degree of difference between the optimized models. In some embodiments, the method can identify the mismatch between the HM and the reference model (GTM) based on a series of graph editing operations in the entire GED calculation. This error backtracking function can also benefit the LLM problem-solving process for mathematical modeling tasks.

[0084] As described herein, a method (including a system architecture) for evaluating the modeling capability of an LLM for an optimization problem (e.g., LPWP) can be based on a graph edit distance. The method can include: converting the LP optimization problem into a general form of an optimization problem. In one embodiment, wherein,

[0085] x=(x1,x2,…x n ) T yes A variable vector in space,

[0086] c=(c1,c2,…c n ) T yes The cost vector in space,

[0087] yes The constraint matrix in space,

[0088] The general form of an optimization (minimization or maximization) linear programming problem can be written as:

[0089]

[0090] Constraints

[0091] where c T is the transpose of the cost vector, l s and u s are the lower and upper bounds of Ax, respectively, and l x and u x are the lower and upper bounds of x respectively.

[0092] also, and is the extended field of real numbers. and are the lower and upper bounds of the decision variable x, and are the lower and upper bounds of the constraint. The types of constraints can include equality, and double-sided or single-sided inequalities. For single-sided inequality constraints, the right-hand side inequality can be prioritized over the left-hand side inequality by multiplying by a constant –1. Also, minimization problems can be prioritized over maximization problems by multiplying by a constant –1.

[0093] The described general form of the LP can be consistent with the input formula of many LP solvers including HiGHS (Huangfu, Q. and Hall, JJ (2018), Parallelization of the Dual Modified Simplex Method, Mathematical Programming Computation, 119-142), CPLEX (ILOG, I. (2010), CPLEX User Manual, http: / / www.ilog.com / ), Gurobi (Gurobi Optimization, L. (2021), Gurobi Optimizer Reference Manual) and other commercial solvers. In one embodiment, a robust parser can be used to extract relevant information (e.g., objectives, variables, and constraints) and establish a general form of the LP.

[0094] One or more other linear programming general forms may also be used to convert the optimization model (hypothesis model and reference model) into the linear programming general form.

[0095] The method for evaluating the modeling ability of LLM for optimization problems can also include graph representation conversion. The linear programming general forms of the obtained HM and GTM can be converted into their corresponding graph representations. In one embodiment, the graph representation can be an attributed bipartite graph, which can be used to represent optimization problems (e.g., LP problems). The method can include obtaining a graph of the HM and a graph of the GTM. The graph of the HM has HM constraint vertices associated with the HM constraints, HM variable vertices associated with the HM variables, and HM edges connecting the HM constraint vertices to the HM variable vertices. The graph of the GTM has GTM constraint vertices associated with the GTM constraints, GTM variable vertices associated with the GTM variables, and GTM edges connecting the GTM constraint vertices to the GTM variable vertices. The method can also include: transforming the graph of the HM into the graph of the GTM through a series of transformation steps. The total number of transformation steps for transforming the graph of the HM into the graph of the GTM is a measure of the accuracy of the HM. That is, if the transformation steps for transforming the graph of the HM into the graph of the GTM are not required, then the two graphs are the same and the HM is accurate.

[0096] Figure 4 A property bipartite graph G = (S∪X, E) 400 is shown, which can represent a linear programming problem. S is a set of constraint vertices: s = {s1, ...s i …s m}402, X is a variable vertex set: X = {X1, ...X j …X n}404, E is the set of edges connecting the constraint vertex to the variable vertex. j and X i They correspond to the jth constraint vector and the ith variable vector respectively. The graph topology is determined by the constraint matrix A. c, A, l x , l u , l s and l u The information contained in can be represented as vertex and edge attributes, as described for the embodiments shown in this article.

[0097] The constraint vertex set S and the variable vertex set X may be disjoint sets. The constraint vertex set S 402 may correspond to a set of constraints in an optimization problem (e.g., LPWP). The variable vertex set X 404 may correspond to a set of variables (decision variables) in the optimization problem. i can correspond to the pair defining the i-th constraint in,

[0098]

[0099] Symbol X in the variable vertex set j Can be overloaded to represent one or more variable attributes, which may include a lower bound Upper Bound and the target coefficient c j : in,

[0100]

[0101] Since the connected vertex s i With vertex X j The edge set E={E(s i ,x j )} if and only if A ij ≠0, so the topology of the attributed bipartite graph G400 can be determined by the constraint matrix A. The attribute of the edge can be determined by the constraint matrix element a ij definition.

[0102] One advantage of representing the general form of LP as an attributed bipartite graph is that the variables are inherently permutation invariant and the constraints are permutation invariant. This means that even if the order of one or more elements can be changed (for example, one or more of the columns in the decision variables, cost vectors, bound vectors, and constraint matrices are permuted), the two models (for example, HM and its corresponding reference model) are still equivalent. Therefore, the method can be unaffected by changes in the order of related elements or items. After this bipartite graph representation, for any optimization model of the general form of LP problem (P), the LP problem (P) can be transformed into an attributed bipartite graph, that is, (P)→G=(S∪X,E).

[0103] The method for evaluating the modeling ability of LLM for optimization problems may also include graph edit distance (GED) calculation. In some embodiments, GED can be defined by a sequence of basic edit operations with the lowest cost for transforming one graph into another graph. The basic edit operations may involve one or more of vertices and edges. The basic edit operations may include one or more of the following: insertion, deletion, and replacement. For general purposes, all of these operations may be referred to as matching, for example, deleting a vertex may be viewed as matching the vertex with an empty vertex ∈.

[0104] GED can be calculated by various techniques. Once the cost of each matching operation is defined, any appropriate technique (including mature methods) can be used. In some embodiments, the cost of each matching operation can be defined based on the principle that each operation on each number in the hypothetical graph requires 1 unit of cost. For example, given a hypothetical graph G h =(S h ∪X h ,E h ) and reference figure G r =(S r ∪X r ,E r ), constrain the vertex Replace with The cost of is the number of mismatched attributes between the two constrained vertices (#msm), i.e. The same cost method used to replace a constraint vertex can be used to replace a variable vertex. The cost of can be based on the number of its attributes, i.e. Similarly, the cost of inserting a vertex can be equivalent to matching the empty vertex ∈ with the inserted vertex.

[0105] The GED between the hypothesis graph and the reference graph may be obtained by any suitable method for computing the GED. In some embodiments, the distance metric may be normalized to the graph size. Larger hypothesis LP problems with larger graphs tend to produce more errors and increase the edit distance of the graph to the corresponding reference graph. In these embodiments, a relative distance relative to the graph size may be used. Therefore, in some embodiments, the method for evaluating the modeling ability of an LLM for an optimization problem may further include normalizing the edit distance by the graph size.

[0106] Figure 5A and Figure 5B An example of a graph editing path from an HM graph to a GTM graph according to an embodiment is shown. Figure 3 Graph 500 may include a first set of vertices including 502 and 504 representing or corresponding to constraints of prediction 306. The first vertex 502 may correspond to a first constraint of prediction 306: The first constraint vertex 502 has an attribute [–∞,50] T A lower bound of -∞ and an upper bound of 50 may be included. The second vertex 504 may correspond to a second constraint: x2≤2x1, which may be rewritten as The second constraint vertex 504 has an attribute of [–∞,0] T Including a lower bound of -∞ and an upper bound of 0. In this embodiment, the HM graph is based on the HM, and the HM is based on the output of the LLM that has provided the LPWP. The LPWP can be a mathematical application optimization problem. The LLM can output the HM variables, the lower bounds of each corresponding HM variable in the HM variables, and the upper bounds of each corresponding HM variable in the HM variables. The LLM can also output at least one HM function that depends on the HM variables. The at least one function can represent the amount to be optimized according to the mathematical application problem. The LLM can also output one or more HM constraint expressions, which define the HM constraints imposed on the HM variables. Obtaining the graph of the HM can include: forming a HM attribute bipartite graph by generating HM constraint vertices (depending on the HM variables and one or more HM constraint expressions), HM variable vertices (depending on the HM variables, the lower bounds and upper bounds of each corresponding HM variable in the HM variables, and at least one HM function) and connecting the HM constraint vertices to the HM variable vertices. The HM constraint vertices form a first HM vertex set, which is disjoint from a second HM vertex set formed by the HM variable vertices. Obtaining the GTM graph is similar to obtaining the HM graph, and necessary modifications may be made.

[0107] Graph 500 may also include a second set of vertices (vertices 512 and 514) associated with variables x1 and x2 of prediction 306, respectively. The properties of vertex 512 may be based on a lower bound of x1 for the function to be optimized (Z = 2x1 + 4x2) Upper bound of x1 and the coefficient c1 of the x1 component. According to the prediction 306 indicating x1 ≥ 0, x1 is bounded lower by zero and upper by ∞. According to the expression to be maximized, c1 is 2. Therefore, the attribute of vertex 512 is c1 = 2. Vertex 512 can be represented by a vector [0,∞,2] T .

[0108] The properties of vertex 514 can be based on the lower bound of x2 of the function to be optimized (Z = 2x1 + 4x2) Upper bound of x2 and the coefficient c2 of the x2 component. According to the prediction 306 indicating x2 ≥ 0, the lower bound of x2 is zero and the upper bound of x2 is ∞. According to the expression to be maximized, c2 is 4. Therefore, the attribute of vertex 514 is c2 = 4. Vertex 514 can be represented by a vector [0,∞,4] T .

[0109] The value of the edge between vertices can be determined according to the constraint expression, the general form of which is:

[0110]

[0111] The result is:

[0112] The first constraint graph 500,

[0113] Second constraint graph 500 .

[0114] Based on the first constraint, the weight of the edge connecting constraint vertex 502 and variable vertex 512 may be 1. Similarly, based on the first constraint, the weight of the edge connecting constraint vertex 502 and variable vertex 514 may be 1. Based on the second constraint, the weight of the edge connecting constraint vertex 504 and variable vertex 512 may be –2. Similarly, based on the second constraint, the weight of the edge connecting constraint vertex 504 and variable vertex 514 may be 1.

[0115] Graph 560 is also an attributed bipartite graph, which can represent a reference model or Figure 3 The GTM 304 of the ground truth 304 can be shown in FIG560. The graph 560 can include a first set of vertices including 562, 564, and 566 representing or corresponding to constraints of the ground truth 304. The vertex 562 can be based on the first constraint of the ground truth 304: The properties constraining vertex 562 may include a lower bound of -∞ and an upper bound of 50. Vertex 562 may be represented in vector form as [-∞,50] T . The GTM can be obtained from any reliable source of LPWP (e.g., LPWP textbook). The GTM can also be based on an analysis of the LPWP by an LPWP expert. For each corresponding GTM variable, the GTM can have a lower bound and an upper bound. The GTM can also have at least one GTM function that depends on the GTM variables. The at least one GTM function can define an output to be optimized based on a mathematical word problem. The GTM can also have one or more GTM constraint expressions that define GTM constraints imposed on the GTM variables.

[0116] Vertex 564 may be based on the second constraint of ground truth 304: x2≤2x1. The second constraint x2≤2x1 may be rewritten as The properties constraining vertex 564 may include a lower bound of –∞ and an upper bound of 0. Vertex 564 may be represented in vector form as [–∞,0] T .

[0117] Vertex 566 can be based on the third constraint: x1≤x2, which can be rewritten as The properties constraining vertex 566 may include a lower bound of –∞ and an upper bound of 0. Vertex 566 may be represented in vector form as [–∞,0] T .

[0118] Graph 560 may also include sets of variable vertices 572 and 574, which correspond to decision variables x1 and x2, respectively, of ground truth 304. The properties of vertex 572 may be based on the lower bound of x1 for the function to be optimized (Z = 2x1 + 4x2) Upper bound of x1 and x1 The coefficient c1 of the component. According to the prediction 306 indicating x1≥5, the lower bound of x1 is 5 and the upper bound is ∞. According to the expression to be maximized, c1 is 2. Therefore, the attribute of vertex 512 is c1 = 2. Vertex 572 can be represented in vector form as [5,∞,2] T .

[0119] The properties of vertex 574 can be based on the lower bound of x2 of the function to be optimized (Z = 2x1 + 4x2) Upper bound of x2 and the coefficient c2 of the x2 component. According to the prediction 306 indicating x2 ≥ 10, the lower bound of x2 is 10 and the upper bound of x2 is ∞. According to the expression to be maximized, c2 is 4. Therefore, the attribute of vertex 574 is c2 = 4. Vertex 514 can be represented by a vector [0,∞,4] T .

[0120] The values ​​of the edges between the vertices of the graph 560 can be determined according to a constraint expression, the general form of which is:

[0121]

[0122] The result is:

[0123] The first constraint graph 560,

[0124] The second constraint graph 560,

[0125] Third constraint graph 560 .

[0126] Based on the first constraint, the weight of the edge connecting the constraint vertex 562 and the variable vertex 572 may be 1. Similarly, based on the first constraint, the weight of the edge connecting the constraint vertex 562 and the variable vertex 574 may be 1. Based on the second constraint, the weight of the edge connecting the constraint vertex 564 and the variable vertex 572 may be –2. Similarly, based on the second constraint, the weight of the edge connecting the constraint vertex 564 and the variable vertex 574 may be 1. Based on the third constraint, the weight of the edge connecting the constraint vertex 566 and the variable vertex 572 may be 1. Similarly, based on the third constraint, the weight of the edge connecting the constraint vertex 566 and the variable vertex 574 may be –1.

[0127] Determining a graph edit path (a minimal sequence of edit operations) from the hypothetical graph 500 to the reference graph 560 may involve the following: Figure 5B In one embodiment, the graph 500 can be edited by one or more editing operations to obtain the graph 520. For example, a graph with attributes [0,∞,2] T The variable vertex 512 can be represented by having attributes [5,∞,2] T 522 of the reference graph 560, where the variable vertex 522 is equivalent to the variable vertex 572 of the reference graph 560. In one embodiment, the cost of the vertex replacement can be based on the count of attribute changes. For example, replacing the variable vertex 512 with the variable vertex 522 involves changing the lower bound attribute 0 of the variable vertex 514 to the lower bound attribute 5 of the variable vertex 522. Based on the attribute change count of 1, the cost of the change can be 1 unit cost.

[0128] The second editing operation may involve using a T The variable vertex 524 is replaced with attributes [0,∞,4]T 514, where variable vertex 524 is equivalent to variable vertex 574 of reference graph 560. In one embodiment, the cost of the vertex replacement involves changing the lower bound attribute 0 of variable vertex 514 to the lower bound attribute 10 of variable vertex 524. Based on 1 attribute change count, the cost of the change can be 1 unit cost.

[0129] Thus, changing graph 500 to graph 520 requires two edit operations corresponding to 2 unit costs. The next set of edit operations based on reference graph 560 may involve inserting constraint vertex 506 into graph 520 to obtain graph 530. Constraint vertex 506 may be based on and correspond to constraint vertex 566 of reference graph 560. Constraint vertex 506 may have the same properties as constraint vertex 566 [–∞,0] T In one embodiment, the cost of inserting the constrained vertex 506 may be based on the number of attributes of the vertex. Based on the two attributes of the lower bound -∞ and the upper bound 0, the cost of inserting the constrained vertex 506 may be 2 units.

[0130] Thus, changing graph 520 to graph 530 requires one edit operation (vertex insertion) corresponding to 2 unit costs, and changing or transforming graph 500 to graph 530 requires a total of 4 unit costs. Any suitable algorithm may be used to automatically transform an HM graph to a GTM graph. Examples of such algorithms include graph edit distance calculation algorithms, A* algorithms, and depth-first graph edit distance algorithms.

[0131] According to one embodiment, the next set of editing operations based on reference graph 560 may involve inserting edges 552 and 554 to change graph 530 to graph 550, which in this example is equivalent to graph 560. As described herein, constraint vertex 506 corresponds to constraint vertex 566 of reference graph 560. To obtain reference graph 560, edges 552 and 554 are added. According to one embodiment, the cost of inserting an edge may be 1 unit cost. Therefore, the cost of inserting edges 552 and 554 may be 2 unit costs. Therefore, the total unit cost of transforming hypothetical graph 510 into reference graph 560 is 6 unit costs, corresponding to two vertex replacements to obtain graph 520, vertex insertions to obtain graph 530, and two edge insertions to obtain graph 550, which is equivalent to reference graph 560.

[0132] Determining one or more editing operations (or transformation steps) to transform graph 500 into graph 550 may indicate an error generated by the LLM that generated prediction (HM) 306. According to one aspect, an error backtracking function may be provided that may identify one or more mismatches between the HM and the reference model based on a series of graph editing operations. The one or more mismatches may refer to one or more editing operations (or transformation steps) used to transform the HM into the reference model. The transformation steps may include: replacing HM constraint values ​​with different HM constraint values, replacing HM variable values ​​with different HM variable values, adding HM constraint vertices, adding HM variable vertices, deleting HM constraint vertices, deleting HM variable vertices, adding an edge connecting an HM constraint vertex to an HM variable vertex, deleting a second edge connecting a second HM constraint vertex to a second HM variable vertex, and replacing the weight of any edge with a different weight.

[0133] According to one aspect, one or more methods described herein can be implemented or incorporated into a platform as a service to evaluate the model building capabilities of operations research (OR) products that use LLM as a basic component. Before these products are officially released, it is very important to verify the model building capabilities to ensure the optimal performance of the released products and maintain a competitive advantage. In one embodiment, the input to the service can be LP (e.g., Figure 1 In some embodiments, the service may accept as input a certain format of the mathematical form of the model (eg, optimization model 104).

[0134] In some embodiments, the output of the service may be a single numeric score indicating how similar a single model input is to its ground truth. In some embodiments, the output of the service may be an accuracy score indicating the overall performance of the LP modeling product on the full test corpus. In some embodiments, the output may include details (e.g., edit actions) of one or more errors (errors / mistakes) (e.g., a series or sequence of errors) in the model built by the product based on the error backtracking function. In some embodiments, the output may include suggestions on how to fix a problem model built by the product.

[0135] According to one aspect, a testing and analysis tool for linear programming automatic modeling products can be provided. In one embodiment, a test corpus (having LP problems and their ground truth mathematical model form) can be used to evaluate LP modeling products that have model forms established for all problems in the test corpus. Each problem in the test corpus can have a HM and a reference model.

[0136] In one embodiment, the complete corpus can be used to evaluate the LP modeling product based on one or more methods described herein. For example, the method may include obtaining a set of LP general forms of the model form generated by the LP modeling product. The method may also include: converting the set of LP general forms into a graph representation, such as an attributed bipartite graph. The method may also include obtaining a set of GEDs based on a graph representation corresponding to the model form and a graph representation of a reference model. In some embodiments, the testing and analysis tool may analyze and return result statistics, including one or more of the following: exact match rate (graph edit distance ratio = 0), average value of normalized graph edit distance, ratio of HMs whose normalized graph edit distance is within a certain threshold (determined by the user).

[0137] In some embodiments, the test and analysis tool can evaluate the LP modeling product based on a single problem. Therefore, the method can be based on a single HM generated by the LP modeling product and the corresponding reference model. In these embodiments, the test and analysis tool can return one or more of the following: a graph edit distance, a series of graph edit operations recorded based on the calculated graph edit distance.

[0138] Figure 6 A flow chart of an embodiment of a method 600 according to the present invention is shown. The method 600 can be used to evaluate the modeling ability of an LLM for an optimization problem by evaluating an HM generated by an LLM according to a GTM. The method 600 includes: at action 601, obtaining a graph of an HM, and at action 602, obtaining a graph of a GTM. The graph of the HM has HM constraint vertices associated with HM constraints, HM variable vertices associated with HM variables, and HM edges connecting the HM constraint vertices to the HM variable vertices. The graph of the GTM has GTM constraint vertices associated with GTM constraints, GTM variable vertices associated with GTM variables, and GTM edges connecting the GTM constraint vertices to the GTM variable vertices. The method may also include: at action 603, transforming the graph of the HM into a graph of the GTM through a series of transformation steps. The total number of transformation steps for transforming the graph of the HM into the graph of the GTM is a measure of the accuracy of the HM. That is, if no transformation steps are required for transforming the graph of the HM into the graph of the GTM, then the two graphs are the same and the HM is accurate.

[0139] According to one aspect, the reward function can be encoded in a reinforcement learning from human feedback (RLHF) process. Fig. 7AThe process of an embodiment of a method for reinforcement learning based on human feedback according to the present invention is shown. Method 700 is based on a machine learning method that combines reinforcement learning techniques with human feedback to train or fine-tune a machine learning model. RLHF is a technique that trains a "reward model" directly based on human feedback and uses the model as a reward function to optimize the agent's strategy (e.g., LLM) using reinforcement learning (RL) through an optimization method. The reward model is trained before optimizing the strategy (machine learning model) to predict whether a given output is good (high reward) or bad (low reward). RLHF can improve the robustness and exploration of RL agents. In step 701, RLHF includes collecting demonstration data and training a supervised strategy. This step involves generating a modeling problem data set in a self-guided manner. The data is used to fine-tune the strategy (machine learning model) through supervised learning. In step 702, RLHF also includes collecting comparison data and training a reward model. The comparison data of step 702 is based on human feedback, and human feedback is used to build a reward model.

[0140] In step 703, RLHF also includes optimizing the policy (machine language model) against the reward model through reinforcement learning. This step involves sampling new questions from the dataset and generating outputs from the policy. The reward model can then calculate the reward for the output. The reward is used to update the policy using reinforcement learning methods such as proximal policy optimization (PPO).

[0141] In step 702, the RLHF is based on a training reward model that relies on large-scale real human feedback, typically in the form of manually labeled text sorting. The labeling process is time-consuming and labor-intensive. According to one embodiment, the reward model generated in step 702 of the RLHF 700 can be replaced by a reward function that operates based on one or more methods described herein. Therefore, an improved method for training LLMs can be provided for linear programming modeling based on reinforcement learning. Technicians can understand that other methods such as the method described in the paper "Training Language Models to Follow Human Feedback Instructions" by Long Ouyang et al. (36th Conference on Neural Information Processing Systems (NeurIPS2022)) can be adjusted by replacing the human feedback aspect with an embodiment of the present invention.

[0142] Figure 7B A method for training a machine learning model based on reinforcement learning according to an embodiment is shown. Method 750 includes collecting demonstration data and training a supervision strategy (e.g., a machine learning model) at 751. Collecting demonstration data and training a supervision strategy 751 can be similar to collecting demonstration data and training a supervision strategy at step 701 of RLHF method 700.

[0143] Method 750 may also include generating 752 one or more reward signals as feedback. In some embodiments, generating one or more reward signals may be based on method 600. For example, method 600 may operate through a reward function to generate one or more reward signals as feedback for reinforcement learning. In one embodiment, generating one or more reward signals may include obtaining a graph representation (e.g., an attribute bipartite graph) of a prediction of a machine learning model or a HM. The HM may be based on an optimization problem (e.g., LPWP) input to the machine learning model. Generating one or more reward signals may also include obtaining a graph representation (e.g., an attribute bipartite graph) of a GTM or a reference model corresponding to the optimization problem. Generating one or more reward signals may also include obtaining a reward signal indicating the quality of the HM. Obtaining a reward signal may include obtaining a set of edit operations (or transformation steps) for transforming the graph representation of the HM into a graph representation of the reference model. Generating one or more signals may eliminate the need for human feedback that may be required in the RLHF. Therefore, the feedback aspect of method 750 may be performed automatically without the need for human feedback.

[0144] The method 750 may also include optimizing 753 a policy (e.g., a machine learning model) based on one or more reward signals via reinforcement learning.

[0145] As described herein, method 750 can replace human feedback rewards (step 702) to train LLM with one or more generated reward signals 752. In some embodiments, method 750 can be used to train LLM in a manner of reinforcement feedback for task-specific modeling, such as linear programming modeling.

[0146] In some embodiments, the total number of transformation steps used to transform the graph of the HM generated by the LLM into the graph of the GTM can be used to determine the accuracy score of the LLM, and can also be used to modify the LLM according to known techniques. When the LLM is modified, other HMs of the LPWP can be generated by the modified LLM, and the graphs of the other HMs can be transformed into the GTM in other numbers of transformation steps. In other embodiments, the modified LLM can be used to generate other HMs of different LPWPs associated with the corresponding GTM. The graphs of the other HMs can continue to be transformed into the graphs of the corresponding GTM, and new accuracy scores can be obtained. The iterative process can be repeated until a stopping criterion is met. The stopping criterion can be based on a preset accuracy value (e.g., the accuracy of the HM is greater than a preset percentage value (e.g., 99%)), a preset maximum number of iterations, a preset maximum running time value, or any other suitable stopping criterion.

[0147] According to one aspect, the problem handling process can be improved by improving the detection of error categories generated by LLM. For example, the problem handling process can be accelerated by quickly detecting the error categories generated by LLM.

[0148] Existing evaluation metrics (e.g., execution and specification metrics) may only determine how many samples in the test data are correct or incorrect. These metrics do not provide insights into specific problems that current LLM versions often encounter. It is possible that LLM performs well on most test cases but struggles with specific patterns. Manual inspection was required in the past to review all incorrect cases and classify them according to their similarities - a process known as "problem handling."

[0149] According to one aspect, the error backtracking function described herein can identify errors (including recurring errors) made by the LLM when generating the HM. The error backtracking function can also realize automatic detection of these errors or problems. After identifying these errors, data enhancement strategies can be used to generate more of these challenging samples to further fine-tune the LLM. By exposing the LLM to more of these data points and their corresponding correct models, the performance of the LLM in dealing with these specific problems can be enhanced.

[0150] Figure 8The flow of an embodiment of a method for training LLM based on error backtracking according to the present invention is shown. Method 800 includes: using an evaluation method 802 to evaluate a LPWP set with an LLM prediction 801 to obtain one or more errors 803 in the prediction. In some embodiments, the evaluation method 802 may refer to performing error backtracking to determine one or more errors in the LLM prediction. In some embodiments, the evaluation method 802 may be similar to method 600. For example, the evaluation method 802 may include obtaining a graph representation (e.g., an attributed bipartite graph) of the LLM prediction. The evaluation method 802 may also include obtaining a graph representation (e.g., an attributed bipartite graph) of a GTM or reference model corresponding to the LPWP set. The evaluation method 802 may also include obtaining an edit operation set (or transformation step) for transforming the graph representation of the prediction into a graph representation of the reference model. The transformation step may indicate one or more errors in the LLM prediction. For example, the evaluation method 802 may identify that many problem samples have generated the same error of "redundant variables" being isolated from any other constraint vertices 803 through the error backtracking function. The method 800 may also include generating a training data set based on the identified one or more errors. In some embodiments, generating a training data set may include identifying common features 805 in problem samples. The common feature may be that redundant variables are always erroneously introduced in samples containing percentage constraints. In some embodiments, identification of common features 805 in problem samples may be automatic, and any suitable method may be used to identify common features between problem samples. In some embodiments, a human reviewer 804 may examine the problem samples and identify their shared or common features.

[0151] After identifying common features between the problem samples (e.g., percentage constraints), generating the training data set may also include employing one or more data augmentation techniques to generate more of these samples. Employing one or more data augmentation techniques may involve manually or automatically augmenting more of these problems with ground truth values ​​of more of these problems (i.e., with percentage constraints). The method 800 may also include training the LLM 806 on the generated training data set or augmented data set. Training the LLM on the generated training data set may enable the LLM to better manage such scenarios.

[0152] According to one or more aspects, systems, methods, and apparatus for an evaluation strategy based on graph edit distance may be provided to evaluate the capability of an LLM in automatically constructing an optimization model.

[0153] The systems, methods, and apparatus described herein may enable more accurate and robust evaluation methods for automatically constructing optimization models using LLMs, addressing deficiencies of existing methods through permutation invariance and better identification of model exact matches.

[0154] In some embodiments, available information in other forms (e.g., textual interpretation) may be integrated into one or more of the methods, systems, and devices described herein. Figure 1 The optimization model 104, each model component (statement) is followed by an explanation of the model component generated by the LLM. Such textual explanations can be added as another attribute to the corresponding vertex in the attribute bipartite graph of the optimization model. A cost function can be further defined for the added attribute and included in the evaluation metrics described in one or more aspects of this article. Integrating available information in other forms has the potential to achieve more fine-grained evaluation metrics and further align with human perspectives.

[0155] In some embodiments, based on the error backtracking function described herein, one or more errors of the prediction model can be obtained. Although in some embodiments, a score indicating the degree of difference between the optimization models can be obtained, in other embodiments, one or more mismatches or errors (including the location of such mismatches) can be obtained based on a series of graph editing operations in the entire GED calculation. This function of identifying one or more errors can also benefit the problem handling process of LLM training described herein.

[0156] Although some embodiments are described with reference to LPWP, the methods, systems, and apparatus described herein can be extended to other types of optimization problems that can be represented by graphs. Other types of optimization problems can include mixed integer linear programming (MILP), quadratic programming (QP), and quadratic programming with quadratic constraints.

[0157] According to the present invention, Fig. 9 An embodiment of an apparatus 900 according to different aspects of the present invention is shown, and the apparatus 900 can perform any or all operations of the above methods and features explicitly or implicitly described herein. For example, a computer equipped with network functions can be configured as the apparatus 900. In some aspects, the apparatus 900 can be a device connected to a network infrastructure via a wireless interface, such as a mobile phone, a smart phone, or other device that can be classified as a user equipment (UE). In some aspects, the apparatus 900 can be a machine type communication (MTC) device (also known as a machine-to-machine (m2m) device) or another such device that can be classified as a UE although it does not provide direct services to users. In some aspects, the apparatus 900 can be used to implement one or more methods, operations, components, systems, mechanisms according to one or more aspects described herein.

[0158] like Fig. 9 As shown, the device 900 may include a processor 910 (e.g., a central processing unit (CPU), or a dedicated processor such as a graphics processing unit (GPU), or other such processor units), a memory 920, a non-transitory mass storage 930, an input-output interface 940, a network interface 950, and a transceiver 960, all of which are communicatively coupled via a bidirectional bus 970. The transceiver 960 may include one or more antennas. According to certain aspects, any or all of the elements described may be utilized, or only a subset of the elements may be utilized. In addition, the device 900 may include multiple instances of certain elements, such as multiple processors, multiple memories, or multiple transceivers. In addition, elements of the hardware device may be directly coupled to other elements without a bidirectional bus. In addition, or in lieu of a processor and memory, other electronic components or processing electronic components (e.g., integrated circuits, application specific integrated circuits, field programmable gate arrays, digital circuits, analog circuits, chips, dies, multi-chip modules, substrates, etc., or combinations thereof) may be used to perform the required logical operations.

[0159] The memory 920 may include any type of non-transient memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or any combination thereof. The mass storage element 930 may include any type of non-transient storage device, such as a solid-state drive, a hard disk drive, a disk drive, an optical drive, a USB drive, or any computer program product for storing data and machine executable program code. According to certain aspects, the memory 920 or the mass storage device 930 may record thereon statements and instructions executable by the processor 910 for performing any method operation described herein.

[0160] The processor 910 and the memory 920 may be used together as a chipset, which may be provided together for installation in the wireless communication device 900 to implement the WLAN function. The chipset may be used to receive data including but not limited to PPDU as input from the network interface 950. The chipset may be used to output data including but not limited to PPDU to the network interface 950.

[0161] Various aspects of the present invention may be implemented using electronic hardware, software, or a combination thereof. In some aspects, this may be implemented by one or more computer processors executing program instructions stored in a memory. In some aspects, for example, the present invention may be implemented in part or in whole in hardware by rapidly performing processing operations via one or more field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs).

[0162] It should be understood that although specific embodiments of the technology have been described herein for illustrative purposes, various modifications may be made without departing from the scope of the technology. Therefore, the specification and drawings are to be regarded only as an illustration of the invention as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations or equivalents within the scope of the invention. Specifically, a computer program product or program element for storing machine-readable signals, or a program storage or memory device such as a magnetic wire, tape, disk or light, optical tape or optical disk is provided, within the scope of the technology, for controlling the operation of a computer according to the method of the technology and / or constructing some or all of its components according to the system of the technology.

[0163] The actions associated with the methods described herein may be implemented as coded instructions in a computer program product. In other words, a computer program product is a computer-readable medium on which software codes are recorded to perform the methods when the computer program product is loaded into a memory and executed on a microprocessor of a wireless communication device.

[0164] In addition, each operation of the method can be performed on any computing device (e.g., personal computer, server, PDA), and is performed according to one or more program elements, modules or objects generated from any programming language (e.g., C++, Java) or a part of one or more program elements, modules or objects. In addition, each operation or a file or object that implements each of the operations, etc., can be performed by dedicated hardware or a circuit module designed for this purpose.

[0165] Through the description of the above embodiments, the present invention can be implemented only by hardware, or by software and necessary general hardware platforms. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile or non-transient storage medium, and the non-volatile or non-transient storage medium can be a compact disk read-only memory (CD-ROM), a USB flash drive, or a removable hard disk. The software product includes many instructions that enable a computer device (a personal computer, a server, or a network device) to perform the method provided in an embodiment of the present invention. For example, such execution may correspond to the simulation of the logical operation described herein. According to an embodiment of the present invention, the software product may additionally or alternatively include a plurality of instructions that enable a computer device to perform the operation of configuring or programming a digital logic device.

[0166] Although combinations of features are shown in the illustrated embodiments, it is not necessary to combine all of the features to achieve the advantages of the various embodiments of the present invention. In other words, a system or method designed according to an embodiment of the present invention does not necessarily include all of the features shown in any one of the drawings or in all of the parts schematically shown in the drawings. In addition, selected features of an exemplary embodiment may be combined with selected features of other exemplary embodiments.

[0167] When used in conjunction with the term "comprising" in the claims and / or the specification, the word "a" may mean "one", but it is also consistent with the meaning of "one or more", "at least one", and "one or more than one", unless the content clearly indicates otherwise. Similarly, the word "another" may mean at least a second or more, unless the content clearly indicates otherwise.

[0168] As used herein, the terms "coupled" or "connected" may have several different meanings depending on the context in which the terms are used. For example, as used herein, the terms "coupled" or "connected" may mean that two elements or devices are connected to each other directly or to each other via mechanical elements through one or more intermediate elements or devices, depending on the specific context. The terms "and / or" herein, when used in conjunction with a list of items, means to include any one or more of the items in the list.

[0169] Although the present invention has been described with reference to specific features and embodiments of the present invention, it is apparent that various modifications and combinations may be made to the present invention without departing from the present invention. Therefore, the specification and drawings are only to be regarded as an illustration of the present invention as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention.

Claims

1. A computer-implemented method for evaluating the accuracy of a hypothesis model (HM) relative to a ground truth model (GTM), characterized in that The method comprises: obtaining a graph of the HM, wherein the graph of the HM comprises HM constraint vertices associated with HM constraints, HM variable vertices associated with HM variables, and HM edges connecting the HM constraint vertices to the HM variable vertices; Obtaining a graph of the GTM, wherein the graph of the GTM comprises GTM constraint vertices associated with GTM constraints, GTM variable vertices associated with GTM variables, and GTM edges connecting the GTM constraint vertices to the GTM variable vertices; The graph of the HM is transformed into the graph of the GTM by a series of transformation steps, the total number of transformation steps used to transform the graph of the HM into the graph of the GTM being a measure of the accuracy of the HM.

2. The computer-implemented method of claim 1, wherein: Transforming the graph of the HM into the graph of the GTM includes: automatically transforming the graph of the HM into the graph of the GTM using a graph edit distance calculation algorithm.

3. The computer-implemented method of claim 1, wherein: Also includes: The HM is obtained by inputting a math word problem into a large language model (LLM) to generate the HM.

4. The computer-implemented method of claim 3, wherein: Inputting the mathematical word problem into the LLM includes: inputting a mathematical application optimization problem into the LLM.

5. The computer-implemented method of claim 3, wherein: Generating the HM includes: Taken from the LLM: The HM variables; a lower bound for each corresponding one of the HM variables; an upper bound for each corresponding one of the HM variables; at least one HM function dependent on the HM variables, wherein the at least one HM function defines an output to be optimized according to the mathematical word problem; one or more HM constraint expressions, wherein the one or more HM constraint expressions define the HM constraints imposed on the HM variables; The at least one HM function and the one or more HM constraint expressions are transformed into a general linear programming model of the mathematical word problem to obtain the HM.

6. The computer-implemented method of claim 5, wherein: The GTM is associated with: a lower bound and an upper bound for each corresponding one of the GTM variables; at least one GTM function dependent on the GTM variable, wherein the at least one GTM function defines an output to be optimized based on the mathematical word problem; One or more GTM constraint expressions, wherein the one or more GTM constraint expressions define the GTM constraints imposed on the GTM variables.

7. The computer-implemented method of claim 6, wherein: Obtaining the graph of the HM includes forming an HM attribute bipartite graph by the following operations: The HM constraint vertices are generated according to the following: The HM variables; The one or more HM constraint expressions; generating the HM variable vertices according to the following: The HM variables; the lower bound and the upper bound for each respective one of the HM variables; the at least one HM function; generating the HM edge according to the one or more HM constraint expressions, wherein the HM edge connects the HM constraint vertex to the HM variable vertex, The HM constraint vertices form a first HM vertex set, The HM variable vertices form a second HM vertex set, The first HM vertex set and the second HM vertex set are disjoint, Obtaining the graph of the GTM includes forming a GTM attribute bipartite graph by the following operations: The GTM constraint vertices are generated according to the following: The GTM variables; The one or more GTM constraint expressions; generating the GTM variable vertex according to the following items: The GTM variables; the lower bound and the upper bound of each corresponding one of the GTM variables; the at least one GTM function; generating the GTM edge according to the one or more GTM constraint expressions, wherein the GTM edge connects the GTM constraint vertex to the GTM variable vertex, The GTM constraint vertices form a first GTM vertex set, The GTM variable vertices form a second GTM vertex set, The first GTM vertex set and the second GTM vertex set are disjoint.

8. The computer-implemented method of claim 1, wherein: The HM constraint vertex is associated with an HM constraint value, The HM variable vertices are associated with HM variable values, The series of transformation steps includes at least one of the following: Replace HM constraint values ​​with different HM constraint values; Replace HM variable values ​​with different HM variable values; Add HM constraint vertices; Add HM variable vertex; Delete HM constraint vertex; Delete HM variable vertex; Add edges connecting the HM constraint vertices to the HM variable vertices; Delete the second edge connecting the second HM constraint vertex to the second HM variable vertex; Replace the weight of any edge with a different weight.

9. The computer-implemented method of claim 3, wherein: Also includes: obtaining an accuracy score based on the total number of the transforming steps, wherein the accuracy score indicates an ability of the LLM to accurately generate the HM; modifying the LLM based on the accuracy score; Actions (A) to (G) are iteratively performed until a stopping criterion is met, and actions (A) to (G) are as follows: (A) generating other HMs by inputting other math word problems into the LLM, wherein the other math word problems are associated with other GTMs; (B) obtaining a map of said other HM; (C) obtaining a graph of said other GTM; (D) transforming said graph of said other HM into said graph of said other GTM by a series of further transformation steps, the total number of said further transformation steps for transforming said graph of said other HM into said graph of said other GTM being a measure of said accuracy of said other HM; (E) obtaining a further accuracy score based on the total number of the further transformation steps, wherein the further accuracy score indicates an ability of the LLM to accurately generate the further HM; (F) modifying the LLM based on the accuracy score; (G) Determining whether the stopping criterion is met.

10. The computer-implemented method of claim 5, wherein: Obtaining a first graph representation comprises: obtaining a first graph representation set of a first optimization model set generated by a machine learning model based on a first optimization problem set, wherein each graph representation in the first graph representation set corresponds to an optimization model in the first optimization model set based on a corresponding optimization problem in the first optimization problem set; Obtaining a second graph representation comprises: obtaining a second graph representation set of a second optimization model set, wherein the second optimization model set is used as a ground truth of the first optimization problem set, and each graph representation in the second graph representation set corresponds to an optimization model in the second optimization model set based on the corresponding optimization problem in the first optimization problem set; Obtaining the minimum number of editing operations includes: obtaining a set of editing operations for transforming the first graph representation set into the second graph representation set.

11. The computer-implemented method of claim 10, wherein: The acquiring of the editing operation set for transforming the first graph representation set into the second graph representation set comprises: For each optimization problem, a sequence of one or more editing operations in the set of editing operations for transforming a corresponding graph representation in the first set of graph representations into a corresponding graph representation in the second set of graph representations is obtained.

12. The computer-implemented method of claim 11, wherein: Also includes: Obtaining a set of graph edit distances (GEDs) corresponding to the set of edit operations, wherein each GED corresponds to each optimization problem and indicates a metric of the corresponding one or more edit operation sequences; Get one or more of the following: an exact match rate based on the GED set, wherein the exact match rate indicates a proportion of the first optimization model set that has a corresponding GED in the GED set that indicates an equivalent match; The average of the GED based on the GED set.

13. The computer-implemented method of claim 10, wherein: Also includes: One or more weights of the machine learning model are adjusted based on the set of editing operations, the machine learning model comprising a plurality of nodes connected to each other via a plurality of connections, the machine learning model also comprising the one or more weights corresponding to the plurality of connections.

14. The computer-implemented method of claim 10, wherein: Also includes: generating a training data set based on the set of editing operations, wherein the training data set includes a second set of optimization problems; The machine learning model is trained using a second training dataset set.

15. A computer-implemented method for evaluating the accuracy of a large language model (LLM), characterized in that: The accuracy lies in generating a hypothesis model (HM) corresponding to a linear programming word problem (LPWP), the method comprising: Obtaining a plurality of HMs corresponding to a plurality of LPWPs from the LLM, wherein each of the plurality of LPWPs is associated with a corresponding ground truth model (GTM), all of the corresponding GTMs form a plurality of GTMs, and each of the plurality of HMs is associated with a corresponding GTM of the plurality of GTMs; For each HM in the plurality of HMs: obtaining a graph of the HM; obtaining a graph of the GTM associated with the HM, the graph of the HM comprises HM constraint vertices associated with HM constraints, HM variable vertices associated with HM variables, and HM edges connecting the HM constraint vertices to the HM variable vertices, The graph of the GTM comprises GTM constraint vertices associated with GTM constraints, GTM variable vertices associated with GTM variables, and GTM edges connecting the GTM constraint vertices to the GTM variable vertices; transforming the graph of the HM into the graph of the GTM through a series of transformation steps having a total number of transformation steps; A score representing the accuracy of the LLM is calculated, wherein the score is a function of the total number of all of the transformation steps used to transform all of the HM graphs associated with the plurality of HMs into all of the corresponding GTM graphs associated with the plurality of GTMs.

16. A method, characterized in that include: Obtain a first graph representation of a hypothetical optimization problem model generated by the machine learning model based on the optimization problem; Obtaining a second graphical representation of a reference optimization problem model based on the optimization problem; Obtain a minimum number of edit operations for transforming the first graph representation to the second graph representation, wherein the minimum number of edit operations indicates performance of the machine learning model.

17. The method according to claim 16, characterized in that The machine learning model is a large language model; The optimization problem is one of the following: Linear programming word problems; Mixed integer linear programming problems; Quadratic programming problems; Quadratic programming problems with quadratic constraints; The first graph representation is a first attributed bipartite graph; The second graph representation is a second attributed bipartite graph; The minimum number of edit operations is a function of a graph edit distance between the first property bipartite graph and the second property bipartite graph.

18. The method according to claim 16, characterized in that The optimization problem includes a set of constraints and a set of decision variables, and the method further includes: Obtaining a first linear programming general form corresponding to the hypothetical optimization problem model; A second linear programming general form corresponding to the reference optimization problem model is obtained.

19. The method according to claim 16, characterized in that The minimum number of editing operations is a minimum sequence of editing operations for transforming the first attributed bipartite graph into the second attributed bipartite graph; The minimal sequence of editing operations is: Involving one or more of the following: vertices in the first vertex set, vertices in the second vertex set, edges in the edge set; Includes one or more of the following: insertion operation, deletion operation, and replacement operation.

20. The method according to claim 16, characterized in that Also includes: Based on the cost value assigned to each editing operation in the minimum editing operation sequence, a total cost of the minimum editing operation sequence is obtained, wherein the total cost indicates the performance of the machine learning model.

Images