Methods and systems for explainable template retrieval for optimization modeling

Abstract

Methods and systems for context-based stepwise generation of optimization models are described. Receiving natural language context information of the optimization problem; updating the structured model summary data structure by retrieving templates of the model component, where each of the one or more templates is retrieved using a respective fine query generated from the contextual information; using the structured model abstract data structure as a query to retrieve a basic model template; updating a symbolic model associated with the base model template by retrieving additional templates of an additional model component, where each of the templates is retrieved using an additional fine query; updating the symbol model using the symbol content associated with each of the retrieved templates; and outputting the symbol model as a generated optimization mode.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202380094839.3 (22) Application Date 2023.02.24 (85) PCT International Application Entering National Phase Date 2025.08.25 (86) PCT International Application Application Data PCT / CN2023 / 078013 2023.02.24 (87) PCT International Application Publication Data WO2024 / 174195 EN 2024.08.29 (71) Applicant Huawei Cloud Computing Technology Co., Ltd. Address 550025 Huawei Cloud Data Center, Xinggong Road, Qianzhong Avenue, Guian New District, Guiyang City, Guizhou Province (72) Inventors: Yu Tianlang, Redra Nirena Ramamungisen Mahdi Mosatabawah, Zhang Yong (74) Patent Agency: Beijing Zhongbo Shida Patent & Trademark Agency Co., Ltd. 11274 Patent Attorney: Shen Jian (51) Int.Cl. G06F 16 / 33 (2006.01) (54) Invention Title: Method and System for Interpretable Template Retrieval for Optimization Modeling (57) Abstract: This invention describes a method and system for progressively generating optimization models based on context. The algorithm receives natural language context information about the optimization problem; updates a structured model summary data structure by retrieving templates of model components, wherein each of one or more templates is retrieved using a corresponding fine-grained query generated from the context information; retrieves a base model template using the structured model summary data structure as a query; updates a symbolic model associated with the base model template by retrieving additional templates of additional model components, wherein each of the templates is retrieved using an additional fine-grained query; updates the symbolic model using symbolic content associated with each of the retrieved templates; and outputs the symbolic model as the generated optimization pattern.Claims 4 pages, Description 27 pages, Drawings 16 pages, CN 121153030 A 2025.12.16 CN 1 21 15 30 30 A 1. A computing system, characterized in that the computing system comprises: a processing unit configured to execute computer-readable instructions to cause the system to perform the following operations: receiving context information from a user device, wherein the context information includes a textual description of an optimization problem in natural language; updating a structured model summary data structure by: retrieving one or more templates of one or more model components from a template database, wherein each module in the one or more templates is retrieved using a fine-grained query generated from the context information; updating the structured model summary data structure using metadata associated with each of the retrieved one or more templates; and retrieving a base model template using the structured model summary data structure as a query to the template database; The symbolic model associated with the base model template is updated by: retrieving one or more additional templates for one or more additional model components from the template database, wherein each of the one or more templates is retrieved using an additional fine-grained query; updating the symbolic model using symbolic content associated with each of the retrieved one or more templates; and outputting the symbolic model as a generated optimized model. 2. The computing system of claim 1, wherein updating the structured model summary data structure includes: updating the context information using metadata associated with the retrieved one or more templates, wherein the updated context information is output. 3. The computing system of claim 1 or 2, wherein updating the structured model summary data structure is performed iteratively, and the structured model summary data structure is updated in each iteration using metadata associated with a retrieved template of a model component. 4. The computing system of claim 3, wherein updating the symbolic model is performed iteratively, and the symbolic model is updated in each iteration using symbolic content associated with a retrieved template of a model component. 5. The computing system according to any one of claims 1 to 4, wherein updating the structured model summary data structure comprises: generating a context-based explanation for each of the retrieved templates (one or more), wherein the context-based explanation is presented via the user device. 6. The computing system according to any one of claims 1 to 5, wherein updating the structured model summary data comprises: presenting the at least one fine-grained query via the user device, and receiving an approved at least one fine-grained query via the user device before retrieving at least one template using the at least one fine-grained query.7. The computing system according to any one of claims 1 to 6, wherein the processing unit is further configured to execute computer-readable instructions to cause the system to perform the following operations: after retrieving the base model template, presenting an explanation of the generation of the base model template through the user device, wherein the generated explanation is generated from the context information and metadata associated with the base model template. 8. The computing system according to any one of claims 1 to 7, wherein updating the symbolic model comprises: presenting the at least one additional fine-grained query through the user device, and receiving an approved at least one additional fine-grained query through the user device before retrieving at least one template using the at least one additional fine-grained query. 9. The computing system according to any one of claims 1 to 8, wherein the at least one fine-grained query is generated based on an initial query received from the user device. 10. The computing system according to any one of claims 1 to 9, wherein the processing unit is further configured to execute computer-readable instructions to cause the system to perform the following operations: providing a user interface (UI) to the user device, wherein the context information is received from the user device as natural language input through the UI. 11. The computing system according to any one of claims 1 to 10, wherein the processing unit is further configured to execute computer-readable instructions to enable the system to implement a fine-grained query generator, wherein the fine-grained query generator comprises: a first natural language processing (NLP) encoder trained to encode context segments segmented from the context information into context vectors and to encode an initial query into a query vector, wherein the context vectors and the query vectors are encoded into a common vector space; a first similarity module configured to select the at least one selected context segment based on the similarity between the query vector and the context vector encoded from at least one selected context segment; and a first NLP generation model trained to generate at least one fine-grained query based on the initial query and the at least one selected context segment.12. The computing system of claim 11, wherein the processing unit is further configured to execute computer-readable instructions to enable the system to implement a template retrieval device, wherein the template retrieval device comprises: a second NLP encoder trained to encode the at least one fine-grained query into a fine-grained query vector and to encode metadata associated with each of the one or more templates into one or more template vectors, wherein the one or more template vectors and the fine-grained query vector are encoded into a common vector space; and a second similarity module configured to select the at least one selected template based on the similarity between the fine-grained query vector and the template vector encoded from the metadata associated with the at least one selected template to be retrieved. 13. The computing system of claim 12, wherein the processing unit is further configured to execute computer-readable instructions to enable the system to implement an interpretation generator, wherein the interpretation generator comprises: a second NLP generative model trained to generate a context-based interpretation for the at least one selected template based on the metadata associated with the at least one selected template and the fine-grained query. 14. A method, characterized in that the method comprises: receiving context information from a user device, wherein the context information includes a textual description of an optimization problem in natural language; updating a structured model summary data structure by: retrieving one or more templates of one or more model components from a template database, wherein each of the one or more templates is retrieved using a corresponding granular query generated from the context information; updating the structured model summary data structure using metadata associated with each of the retrieved one or more templates; retrieving a base model template using the structured model summary data structure as a query to the template database; updating a symbolic model associated with the base model template by: retrieving one or more additional templates of one or more additional model components from the template database, wherein each of the one or more templates is retrieved using an additional granular query; updating the symbolic model using symbolic content associated with each of the retrieved one or more templates; and outputting the symbolic model as a generated optimization model. Claims 2 / 4, page 3, CN 121153030, A 15. The method according to claim 14, wherein updating the structured model summary data structure comprises: updating the context information using metadata associated with the retrieved one or more templates, wherein the updated context information is output.16. The method of claim 14 or 15, wherein updating the structured model summary data structure is performed iteratively, and the structured model summary data structure is updated in each iteration using metadata associated with a retrieved template of a model component. 17. The method of claim 16, wherein updating the symbolic model is performed iteratively, and the symbolic model is updated in each iteration using symbolic content associated with a retrieved template of a model component. 18. The method of any one of claims 14 to 17, wherein updating the structured model summary data structure includes: generating a context-based explanation for each of the retrieved one or more templates, wherein the context-based explanation is presented via the user device. 19. The method of any one of claims 14 to 18, wherein updating the structured model summary data includes: presenting the at least one fine-grained query via the user device, and receiving an approved at least one fine-grained query via the user device before retrieving at least one template using the at least one fine-grained query. 20. The method of any one of claims 14 to 19, further comprising: after retrieving the base model template, presenting an explanation of the generation of the base model template via the user device, wherein the explanation is generated from the context information and metadata associated with the base model template. 21. The method of any one of claims 14 to 20, wherein updating the symbolic model comprises: presenting the at least one additional fine-grained query via the user device, and receiving an approved at least one additional fine-grained query via the user device before retrieving at least one template using the at least one additional fine-grained query. 22. The method of any one of claims 14 to 21, wherein the at least one fine-grained query is generated based on an initial query received from the user device. 23. The method of any one of claims 14 to 22, further comprising: providing a user interface (UI) to the user device, wherein the context information is received from the user device as natural language input via the UI.24. The method according to any one of claims 14 to 23, characterized in that the method further comprises executing a fine-grained query generator, wherein the fine-grained query generator comprises: a first natural language processing (NLP) encoder trained to encode context segments segmented from the context information into context vectors and to encode an initial query into a query vector, wherein the context vectors and the query vectors are encoded into a common vector space; a first similarity module configured to select the at least one selected context segment based on the similarity between the query vector and a context vector encoded from at least one selected context segment; and a first NLP generative model trained to generate at least one fine-grained query based on the initial query and the at least one selected context segment. 25. The method of claim 24, further comprising executing a template retrieval unit, wherein the template retrieval unit comprises: a second NLP encoder trained to encode the at least one fine-grained query into a fine-grained query vector and to encode metadata associated with each of one or more templates into one or more template vectors, wherein the one or more template vectors and the fine-grained query vector are encoded into a common vector space; and a second similarity module for selecting the at least one selected template based on the similarity between the fine-grained query vector and template vectors encoded from metadata associated with the at least one selected template to be retrieved. 26. The method of claim 25, further comprising executing an interpretation generator, wherein the interpretation generator comprises: a second NLP generative model trained to generate a context-based interpretation for the at least one selected template based on the metadata associated with the at least one selected template and the fine-grained query. 27. A non-transitory computer-readable medium storing instructions, characterized in that the instructions are executable by a processing unit in a computing system to cause the computing system to perform the method according to any one of claims 14 to 26. Claims 4 / 4 Page 5 CN 121153030 A Method and System for Interpretable Template Retrieval for Optimization Modeling Technical Field

[0001] The present invention relates to a method and system for generating a model for solving an optimization problem using an interpretable template retrieval system. Background Art

[0002] Many problems in the real world are optimization problems, such as how to minimize the use of limited resources or how to maximize system throughput.In many cases, these optimization problems are so complex (e.g., with many different constraints, many different conditions, many different variables, etc.) that they are practically unsolvable by humans. Optimization solvers (also simply “solvers”) are software tools developed to solve such optimization problems. To enable the solver to understand the optimization problem, it is constructed as an optimization model. This optimization model includes a mathematical representation of one or more decision variables, one or more constraints, and one or more objectives of the optimization problem.

[0003] The conventional process of generating optimization models typically involves: a person familiar with the optimization problem and its context (e.g., a business owner) first describing the optimization problem in a non-technical manner, for example, in a natural language document (referred to herein as a problem specification); then, another person (e.g., an operations research (OR) expert) converts the problem specification into an optimization model and instantiates the model as a model instance using actual numerical data, wherein the model instance is in the form of computer-readable code that can be input into the solver. However, the traditional process of generating optimization models typically requires repeated questioning, modeling, and verification by both non-OR experts (e.g., the person posing the optimization problem) and OR experts to obtain a final optimization model that satisfies both. This process can be inefficient in terms of both time and resources. Since OR experts can become a bottleneck, the demand for OR experts also limits the practical use of the solver.

[0004] Ideally, a technical solution should be provided that can improve the accessibility of the solver and / or improve the efficiency of optimization problem modeling. Summary of the Invention

[0005] In various examples, the present invention describes an interpretable template retrieval system capable of generating optimization models from non-technical natural language documents describing optimization problems. The interpretable template retrieval system may include subsystems such as a query optimizer, a template retriever, and an explanation generator.

[0006] The methods and systems disclosed in this invention can retrieve templates based on problem specifications that provide a natural language text description of the optimization problem. The methods and systems disclosed in this invention can also generate explanations for the retrieved templates, wherein the explanations are based on the context of the problem description. The advantage of this approach is that the interpretation of the retrieved templates can provide information to help understand the templates within the context of a specific problem, rather than a general description of the templates.

[0007] Various examples of the present invention can use machine learning to process natural language problem specifications and help human users build optimization models. The advantage of this is that non-technical users can generate optimization models for specific optimization problems without manually creating code or mathematical formulas to represent the optimization problem. Users can build incrementally on existing model templates to generate optimization models tailored to specific optimization problems.

[0008] In one exemplary aspect, the present invention describes a computing system. The computing system includes a processing unit, as described on page 1 / 27 of the specification (CN 121153030 A). The processing unit is configured to execute computer-readable instructions to cause the system to perform the following operations: receiving context information from a user device, wherein the context information includes a textual description of an optimization problem in natural language; updating a structured model summary data structure by retrieving one or more templates of one or more model components from a template database, wherein each of the one or more templates is retrieved using a corresponding granular query generated from the context information; updating the structured model summary data structure using metadata associated with each of the retrieved one or more templates; retrieving a base model template using the structured model summary data structure as a query to the template database; updating a symbolic model associated with the base model template by retrieving one or more additional templates of one or more additional model components from the template database, wherein each of the one or more templates is retrieved using an additional granular query; updating the symbolic model using symbolic content associated with each of the retrieved one or more templates; and outputting the symbolic model as a generated optimization model.

[0009] In one example of the above exemplary aspect of the system, updating the structured model summary data structure may include: updating the context information using metadata associated with the retrieved one or more templates, wherein the updated context information is output.

[0010] In one example of any of the above exemplary aspects of the system, updating the structured model summary data structure may be performed iteratively, wherein the structured model summary data structure may be updated in each iteration using metadata associated with a retrieved template of a model component.

[0011] In one example of any of the above exemplary aspects of the system, updating the symbolic model may be performed iteratively, wherein the symbolic model may be updated in each iteration using symbolic content associated with a retrieved template of a model component.

[0012] In one example of any of the above exemplary aspects of the system, updating the structured model summary data structure may include: generating a context-based explanation for each of the retrieved one or more templates; the context-based explanation may be presented via the user device.

[0013] In one example of any of the above exemplary aspects of the system, updating the structured model summary data may include: presenting the at least one fine-grained query through the user device, and receiving the approved at least one fine-grained query through the user device before retrieving at least one template using the at least one fine-grained query.

[0014] In one example of any of the above exemplary aspects of the system, the processing unit may further be configured to execute computer-readable instructions to cause the system to perform the following operations: after retrieving the base model template, presenting an explanation of the generation of the base model template through the user device, wherein the generated explanation is generated from the context information and metadata associated with the base model template.

[0015] In one example of any of the above exemplary aspects of the system, updating the symbolic model may include: presenting the at least one additional fine-grained query through the user device, and receiving an approved at least one additional fine-grained query through the user device before retrieving at least one template using the at least one additional fine-grained query.

[0016] In one example of any of the above exemplary aspects of the system, the at least one fine-grained query may be generated based on an initial query received from the user device.

[0017] In one example of any of the above exemplary aspects of the system, the processing unit may further be configured to execute computer-readable instructions to cause the system to perform the following operations: providing a user interface (UI) to the user device, wherein the context information is received from the user device as natural language input through the UI.

[0018] In one example of any of the above exemplary aspects of the system, the processing unit may also be configured to execute computer-readable instructions as per page 2 / 27 of the specification, CN 121153030 A, to enable the system to implement a fine-grained query generator, wherein the fine-grained query generator comprises: a first natural language processing (NLP) encoder trained to encode context fragments segmented from the context information into context vectors and to encode an initial query into a query vector, wherein the context vectors and the query vectors are encoded into a common vector space; a first similarity module configured to select the at least one selected context fragment based on the similarity between the query vector and the context vector encoded from at least one selected context fragment; and a first NLP generation model trained to generate at least one fine-grained query based on the initial query and the at least one selected context fragment.

[0019] In one example of any of the above exemplary aspects of the system, the processing unit may further be configured to execute computer-readable instructions to enable the system to implement a template retrieval device, wherein the template retrieval device includes: a second NLP encoder trained to encode the at least one fine-grained query into a fine-grained query vector and to encode metadata associated with each of the one or more templates into one or more template vectors, wherein the one or more template vectors and the fine-grained query vectors are encoded into a common vector space; and a second similarity module configured to select the at least one selected template based on the similarity between the fine-grained query vector and the template vector encoded from the metadata associated with the at least one selected template to be retrieved.

[0020] In one example of any of the above exemplary aspects of the system, the processing unit may further be configured to execute computer-readable instructions to enable the system to implement an interpretation generator, wherein the interpretation generator includes: a second NLP generative model trained to generate a context-based interpretation for the at least one selected template based on the metadata associated with the at least one selected template and the fine-grained query.

[0021] In one exemplary aspect, the present invention describes a method. The method includes: receiving context information from a user device, wherein the context information includes a textual description of an optimization problem in natural language; updating a structured model summary data structure by retrieving one or more templates of one or more model components from a template database, wherein each of the one or more templates is retrieved using a corresponding fine-grained query generated from the context information; updating the structured model summary data structure using metadata associated with each of the retrieved one or more templates; retrieving a base model template using the structured model summary data structure as a query to the template database; updating a symbolic model associated with the base model template by retrieving one or more additional templates of one or more additional model components from the template database, wherein each of the one or more templates is retrieved using an additional fine-grained query; updating the symbolic model using symbolic content associated with each of the retrieved one or more templates; and outputting the symbolic model as a generated optimization model.

[0022] In one example of any of the above exemplary aspects of the method, updating the structured model summary data structure may include: updating the context information using metadata associated with the retrieved one or more templates, wherein the updated context information may be output.

[0023] In one example of any of the above exemplary aspects of the method, the updating of the structured model summary data structure may be performed iteratively, and the structured model summary data structure may be updated in each iteration using metadata associated with a retrieved template of a model component.

[0024] In one example of any of the above exemplary aspects of the method, the updating of the symbolic model may be performed iteratively, and the symbolic model may be updated in each iteration using symbolic content associated with a retrieved template of a model component.

[0025] In one example of any of the above exemplary aspects of the method, the updating of the structured model summary data structure may include: generating a context-based explanation for each of the retrieved one or more templates; the context-based explanation may be presented through the user device.

[0026] In one example of any of the above exemplary aspects of the method, the updating of the structured model summary data may include: presenting the at least one fine-grained query through the user device, and receiving an approved at least one fine-grained query through the user device before retrieving at least one template using the at least one fine-grained query.

[0027] In one example of any of the above exemplary aspects of the method, the method may include: after retrieving the base model template, presenting an explanation of the generation of the base model template via the user device, wherein the generated explanation is generated from the context information and metadata associated with the base model template.

[0028] In one example of any of the above exemplary aspects of the method, updating the symbolic model may include: presenting the at least one additional fine-grained query via the user device, and receiving an approved at least one additional fine-grained query via the user device before retrieving at least one template using the at least one additional fine-grained query.

[0029] In one example of any of the above exemplary aspects of the method, the at least one fine-grained query may be generated based on an initial query received from the user device.

[0030] In one example of any of the above exemplary aspects of the method, the method may include: providing a user interface (UI) to the user device, wherein the context information is received from the user device as natural language input via the UI.

[0031] In one example of any of the above exemplary aspects of the method, the method may include executing a fine-grained query generator, wherein the fine-grained query generator includes: a first natural language processing (NLP) encoder trained to encode context fragments segmented from the context information into context vectors and to encode an initial query into a query vector, wherein the context vectors and the query vectors are encoded into a common vector space; a first similarity module configured to select the at least one selected context fragment based on the similarity between the query vector and the context vector encoded from at least one selected context fragment; and a first NLP generative model trained to generate at least one fine-grained query based on the initial query and the at least one selected context fragment.

[0032] In one example of any of the above exemplary aspects of the method, the method may include an execution template retrieval unit, wherein the template retrieval unit includes: a second NLP encoder trained to encode the at least one fine-grained query into a fine-grained query vector and to encode metadata associated with each of the one or more templates into one or more template vectors, wherein the one or more template vectors and the fine-grained query vector are encoded into a common vector space; and a second similarity module for selecting the at least one selected template based on the similarity between the fine-grained query vector and the template vector encoded from the metadata associated with the at least one selected template to be retrieved.

[0033] In one example of any of the above exemplary aspects of the method, the method may include an execution interpretation generator, wherein the interpretation generator includes: a second NLP generative model trained to generate a context-based interpretation for the at least one selected template based on the metadata associated with the at least one selected template and the fine-grained query.

[0034] In another exemplary aspect, the present invention describes a non-transitory computer-readable medium storing machine-executable instructions. When the instruction is executed by the processing unit in the device, the device performs the method described in any of the above examples.Specification 4 / 27 Page 9 CN 121153030 A Brief Description of the Drawings

[0035] The following will illustrate exemplary embodiments of the present application by way of example with reference to the accompanying drawings, in which:

[0036] FIG1 is a block diagram of an exemplary interpretable template retrieval system provided by various embodiments of the present invention;

[0037] FIG2 is a block diagram of an exemplary computing system that can be used to implement various embodiments of the present invention;

[0038] FIG3A is a block diagram of components in an exemplary fine-grained query generator that can be used in the system shown in FIG1, provided by various embodiments of the present invention;

[0039] FIG3B shows an example of how the fine-grained query generator shown in FIG3A provided by various embodiments of the present invention can generate fine-grained queries;

[0040] FIG4A is a block diagram of components in an exemplary template retriever that can be used in the system shown in FIG1, provided by various embodiments of the present invention;

[0041] FIG4B shows an example of how the template retriever shown in FIG4A provided by various embodiments of the present invention can retrieve templates;

[0042] FIG5A is a block diagram of components in an exemplary interpretable generator that can be used in the system shown in FIG1, provided by various embodiments of the present invention;

[0043] Figure 5B shows an example of how the explanation generator shown in Figure 5A, provided by various examples of the present invention, can generate an explanation;

[0044] Figure 6 is a flowchart of an exemplary incremental modeling method provided by various examples of the present invention;

[0045] Figures 7 to 9 are flowcharts of examples of steps for implementing the incremental modeling method shown in Figure 6, provided by various examples of the present invention;

[0046] Figures 10A to 10G show exemplary user interfaces that can be provided by various examples of the present invention.

[0047] Similar reference numerals may be used to denote similar components in different figures. Detailed Description

[0048] To aid in understanding the present invention, some background discussion of optimization problems and optimization models is first provided.

[0049] Generally, an optimization problem describes a decision scenario involving a goal that needs to be minimized or maximized. The problem specification defines the optimization problem in natural language. Problem specifications are often created by domain experts who understand the context of the problem but may not be able to express the problem mathematically or technically. Typically, problem specifications may be ambiguous and / or unstructured. Problem specifications provide the context of the problem and provide background information, which may help clarify ambiguous terms or information. Generally, a problem specification outlines one or more objectives to be optimized, decision variables adjusted to optimize one or more objectives, symbolic parameters defining the parameters of one or more objectives, and one or more constraints of the problem. However, a problem specification may describe these components of an optimization problem in a vague way, which only makes sense in context.For example, a problem specification may include a natural language phrase, such as “costs should be minimized,” which is vague and lacks context. The context of this phrase might be that “costs” refers to the hourly wage cost of drivers and is affected by the driving time required by each driver.

[0050] In order for an optimization solver (hereinafter referred to as a “solver” or “software solver”) to understand and solve the optimization problem, the optimization problem must be constructed as an optimization model. The optimization model includes a symbolic model and semantic metadata. The symbolic model is a symbolic and context-independent representation of the optimization problem, typically expressed using a modeling language or programming language (i.e., a non-natural human language). The symbolic model may mathematically represent one or more objectives of the optimization problem (the symbolic model may be represented as a mathematical function to be optimized and may be parameterized by symbolic parameters), may include mathematical representations of constraints (e.g., equality constraints or inequality constraints), and may include model entities (e.g., indexed decision variables or parameters). Semantic metadata provides information describing the meaning of the mathematical representations in the symbolic model. For example, semantic metadata may include a target name that gives meaning to the objective function (e.g., "maximize the output"), constraint attributes that give meaning to constraints (e.g., the right-hand side expression), or model attributes that provide information about the model name and the number of constraint variables, etc.

[0051] It should be noted that the optimization model may need to be instantiated with actual data values ​​in order to be solved by the solver. When the data parameters in the symbolic model are given actual data values ​​(e.g., from a scene in the real world), the symbolic model is instantiated, creating a model instance. This model instance is a specific instance of the symbolic model in the form of computer-readable code, which can be input into the solver to generate a solution to the optimization problem. There are various solvers available for solving different optimization problems, and each solver may need to be provided with different specific structured inputs by the model instance (e.g., specific keywords, specific code order, etc.).

[0052] Generating an optimization model usually requires the skills of an operations research (OR) expert to convert the natural language problem specification into an optimization model (e.g., converting the objective from natural language into an optimizable mathematical function). However, domain experts who are not OR experts may find it difficult to understand the optimization model. Therefore, domain experts may find it difficult to verify whether the optimization model accurately represents the optimization problem, and thus it is difficult to verify and utilize the optimization model. Similarly, without the help of OR experts, domain experts also find it difficult to make even small changes to the optimization model (e.g., adjusting decision variables or parameters).

[0053] There is great interest in using machine learning to help improve the efficiency of the optimization model generation process.Some existing technical solutions help users generate an optimization model by guiding them to answer a series of questions, thereby obtaining a base model, which can then be adjusted by adding additional model components. However, the guided questioning process can be restrictive and unintuitive for users. There is also interest in using natural language processing (NLP) machine learning models that accept natural language input but not programming languages. For example, existing technical solutions include interpretable information retrieval systems, which are systems that perform document retrieval in response to queries and are designed to provide an explanation of how the retrieved documents are relevant to the query. However, many existing interpretable information retrieval systems may not be able to refine the query using contextual information or provide an explanation suitable for the user's context.

[0054] In general, it is desirable to have a technical solution that can accept a natural language description of the optimization problem, generate a suitable optimization model, and provide a context-based explanation of the model to help address some or all of the shortcomings of existing technical solutions. In various examples, the methods and systems described in this invention are able to generate optimization models according to natural language problem specifications and provide context-based explanations to validate the model and improve the efficiency of model generation. This invention discloses an interpretable template retrieval system and a method for generating optimization models using the interpretable template retrieval system within an incremental modeling framework.

[0055] Various examples of this invention can be implemented as online services, mobile applications, or other computer-aided design software applications. Various examples of this invention allow users to provide natural language input to progressively build optimization models. Context-based explanations can be generated to help explain and guide the model building process.

[0056] Various examples of this invention can provide a more intuitive and flexible model generation method. For example, various examples of this invention can adapt to information provided in the problem specification and enable user interaction to build model components based on a specific optimization problem, rather than restricting users to using predefined decision trees. Various examples of this invention can also provide a user interface that allows users to modify models and model components without requiring users to understand optimization modeling or coding.

[0057] In addition to generating optimization models, various examples of this invention can also generate an annotated version of the optimization specification, which can be used as documentation for the optimization model and the building process. In this way, the transparency of the optimization model can be improved, and manual recording of the model can be avoided.

[0058] Figure 1 is a schematic diagram of an exemplary interpretable template retrieval system 100.The interpretable template retrieval system 100 can be implemented in a single physical machine or device (e.g., as a single computing device, such as a single workstation, a single server, etc.) or using multiple physical machines or devices (e.g., as a server cluster). For example, the interpretable template retrieval system 100 can be implemented as a virtual machine or a cloud-based service (e.g., using a cloud computing platform that provides a pool of virtualized computing resources). In some examples, the interpretable template retrieval system 100 can provide a cloud-based service accessible to a user device.

[0059] In this example, the interpretable template retrieval system 100 communicates with the user device 10 and the template database 150 (e.g., via a wired or wireless network). The template database 250 can be external to the interpretable template retrieval system 100 (as shown) or part of the interpretable template retrieval system 100. In some examples, the interpretable template retrieval system 100 can also optionally communicate with a model storage (not shown), which can be a memory for storing optimized models generated by the interpretable template retrieval system 100. In some examples, the interpretable template retrieval system 100 may cooperate with or be part of a model instantiation system (not shown) to instantiate the generated optimization model using data values; the model instance can then be input into a solver (not shown) to solve the optimization problem.

[0060] In the example shown, the interpretable template retrieval system 100 includes a fine query generator 200, a template retrieval unit 300, and an interpretation generator 400 as subsystems. It should be understood that in some embodiments, the interpretable template retrieval system 100 may include more or fewer subsystems than shown in FIG. 1. For example, operations described as being performed by a particular subsystem may be performed by different subsystems in the interpretable template retrieval system 100. As another example, the interpretable template retrieval system 100 itself may provide the functionality of fine query generation, template retrieval, and interpretation generation without including subsystems 200, 300, and 400. It should be understood that the interpretable template retrieval system 100 may be implemented using various hardware and software and is not limited to the example shown in FIG. 1.

[0061] The user may provide contextual information through the user device 10. The context information can be a natural language problem specification describing the optimization problem to be solved. The initial query can also optionally be provided by the user via user device 10, or it can be obtained from a database storing structured model summaries (discussed further below). The initial query is a query for a specific template to be used to generate the optimization model. The fine-grained query generator 200 processes the context information to extract contextual information relevant to the initial query and adds the extracted contextual information to the initial query to generate the fine-grained query.For example, the refined query generator 200 can generate a summary of contextual information and append that summary to the initial query to generate a refined query. As another example, the refined query generator 200 can provide contextual information to clarify ambiguous text in the query. For example, if the query includes pronouns such as “it,” the refined query generator 200 can provide context to define what “it” refers to; or, if the query includes terms such as “cost” that require further definition, the refined query generator 200 can provide context to define that term. As another example, the refined query generator 200 can extract one or more keywords from the contextual information and (e.g., by concatenating one or more keywords into the initial query) incorporate the extracted one or more keywords into the initial query to generate a refined query.

[0062] For example, if the optimization problem relates to the delivery of goods by a delivery company, the contextual information might include information such as the number of drivers and trucks owned by the company, working hour preferences, physical limitations of the trucks, fuel costs, etc. This contextual information can be provided in a relatively long specification (e.g., a manual of over one hundred pages). This contextual information can be written in natural human language. If this contextual information is prepared by the user, it may include the user's relevant style information, OR's expertise, technical terminology, etc. The fine query generator 200 can perform various operations to extract the parts of the contextual information that are relevant to the optimization problem. Depending on the format of the contextual information, the extracted context may also take different forms. For example, the extracted context may be an extracted sentence or text fragment. In other examples, the extracted context may be a list of keywords or a list of named tuples of metadata definitions and values, etc.

[0063] In some examples, the user may input an initial query through the user device 10. The query may be input in natural language and may lack sufficient context. For example, the initial query provided by the user may be "I want my driver to save money", but this does not provide sufficient context affecting "cost". The fine-grained query generator 200 can perform various operations to extract the text portion most relevant to the initial query from the context information (e.g., "costs are related to the driver's driving time on the road, the number of employees per day, and overtime pay"), and concatenate (or append) the initial query with the extracted context information to generate a fine-grained query. For example, the generated fine-grained query might be "I want to minimize driver costs. Driver costs are related to the driver's driving time on the road, the number of employees per day, and overtime pay."

[0064] As another example, the initial query can be obtained from a storage database of structured model summaries and can also be supplemented with information from the context information.For example, the initial query in the database of structured model summaries might be "What key metrics do you want to observe (maximize or minimize)?", which might be further supplemented with "Based on your description, your goal is to reduce overall operating costs." Therefore, information found in the context can be appended to the initial query to provide suggested queries that the user might accept or reject (or modify). If the user accepts this initial query, the fine-grained query generator 200 can perform other operations, extracting additional contextual information related to "cost" in the initial query, similar to the example above. The generated fine-grained query could be "I want to minimize operating costs defined by 'drivers' time on the road, daily number of employees, and overtime pay'".

[0065] The fine-grained query generated by the fine-grained query generator 200 is provided as input to the template retrieval unit 300. The template retrieval unit 300 uses the fine-grained query to retrieve a retrieved template from the template database 150. The template database 150 can store multiple predefined templates associated with model components that can be added to the optimized model. Each template stored in template database 150 has some metadata associated with the template (e.g., metadata about the meaning of the template, such as "minimize total driving time", "minimize driver wages", etc.), and also symbolic content as the mathematical representation of the template (e.g., Min sum(t_{ij…), Min sum(f_{k}y_{k}…) etc.). Therefore, the metadata of each template contains some information related to context, as well as some information about the mathematical content of the template, such as dimensions. The metadata associated with each template is used to identify the template most relevant to the granular query from template database 150. For example, consider the exemplary granular query "I want to minimize operating costs limited by "driver driving time on the road, number of employees per day, and overtime pay". Template database 150 can store the following templates: template T1 with metadata "minimize total driving time" and mathematical content Min sum(t_{ij}x_{ij}); template T2 with metadata "minimize driver wages" and mathematical content Min sum(f_{k}y_{k}+c_{k}w_{k}); and template T3 with metadata "minimize the number of standby drivers" and mathematical content Min sum(y_{k}). Template retrieval unit 300 uses a fine-grained query to find the most relevant template among templates T1, T2, and T3, and retrieves the retrieved template T1.

[0066] Then, the retrieved template and the fine-grained query are input into the explanation generator 400, which generates a generated explanation. The generated explanation can be output for display on user device 10. The generated explanation describes the relevance of the retrieved template to the initial query and context information.The generated explanation uses contextual information from the granular query, generating an explanation based on the language and level of expertise indicated in the contextual information. This means that the generated explanation can be customized based on the language and understanding of the user providing the contextual information (rather than based on a generic explanation predefined for the template). This allows the user to better understand the retrieved template, thereby approving the retrieved template for generating an optimized model.

[0067] It should be noted that the explanation generator 400 generates the explanation from the granular query rather than from the generic initial query. The granular query includes some contextual information provided by the user, enabling the explanation generator 400 to provide an explanation within the context of the user's question. Furthermore, using the granular query as input to the explanation generator 400 is superior to using the entire original context specification, which may be ambiguous, unstructured, and / or verbose, making it difficult to use as the basis for explanation generation. For example, if the fine-grained query is "I want to minimize operating costs defined by "drivers' travel time, daily number of employees, and overtime pay", and the retrieved template is T1 with metadata "minimize total travel time", then the explanation generator 400 can extract the similarity between the fine-grained query and the retrieved template to generate the explanation "You mentioned 'minimize operating costs,' and the cost depends on 'drivers' travel time.' Therefore, by minimizing the total travel time, the cost can be minimized." This generated explanation can be provided to the user through the user device 10.

[0068] The details of the subsystems 200, 300, and 400 in the interpretable template retrieval system 100 are further discussed below, including exemplary NLP language models that can be used to implement subsystems 200, 300, and 400 and exemplary methods for training neural networks for the tasks of each subsystem 200, 300, and 400. In short, a trained NLP model (e.g., a converter model, including encoder-only converter models such as BERT) can be used to encode contextual information, queries, and / or templates into low-dimensional vectors (also called embedded vectors or simply embeddings), and then these vectors are compared (e.g., a cosine similarity metric is computed) to identify and retrieve relevant templates from the template database 150 (i.e., templates relevant to the query based on vector similarity). In some examples, the vectors described in this invention can be considered as or referred to as embedded vectors. It should be understood that the term "vector" as used herein can include "embedded vector" or "embedding".

[0069] The interpretation generator 400 can generate interpretations in various ways.For example, the explanation generator 400 can use a trained NLP generative model (e.g., a trained encoder-decoder generative model, such as the BART model) trained to perform an answering task, where the fine-grained query from the query optimizer 200 is a knowledge base, from which the explanation generator 400 extracts context to answer the question "Why is this template relevant?". Alternatively, the explanation generator 400 can use a trained model trained to perform a retrieval task, where contextual information is segmented into sentences or larger fragments that may correspond to broader viewpoints, and each text fragment can be considered a retrieveable document. An NLP encoder (e.g., an encoder-transformer-only model, such as BERT) is used to convert the retrieved template and each fragment of these text fragments into vectors, and the similarity between these vectors (e.g., based on a computed cosine similarity metric) can be used to retrieve one or more of the most similar fragments. Then, the interpreter generator 400 can use an NLP generative model (e.g., an encoder-decoder converter model, or any generative model such as BART, GPT-3, T5, etc.) to generate a natural language interpretation using metadata from the retrieved template and one or more of the most relevant text fragments retrieved from a fine-grained query or contextual information. The input to the NLP generative model can have various formats. For example, metadata from the retrieved template metadata can be concatenated (e.g., by...). <query>Some text< / query> <metadata> Some text< / metadata>(e.g., delimited by markers), can be combined using a trained summarization model (which takes text fragments as input and generates summaries), or keywords can be extracted from metadata and appended to fine-grained queries, etc.

[0070] As described above, the interpretable template retrieval system 100 can be implemented using different hardware configurations. Exemplary computing systems that can be used to implement the interpretable template retrieval system 100 are described below.

[0071] FIG2 is a block diagram of a simplified exemplary computing system 500 that can be used to implement the interpretable template retrieval system 100 in some embodiments. The computing system 500 can represent a server or workstation, etc. As described above, the interpretable template retrieval system 100 can be implemented in other hardware configurations, including using multiple computing systems. Although FIG2 shows a single instance of each component, multiple instances of each component can exist in the computing system 500. The computing system 500 can be used to execute instructions for training neural networks and / or for executing trained neural networks, as discussed further below.

[0072] In this example, the computing system 500 includes at least one processing unit 502. At least one processing unit 502 may be a processor, microprocessor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), dedicated logic circuit, dedicated artificial intelligence processing unit, graphics processing unit (GPU), tensor processing unit (TPU), neural processing unit (NPU), hardware accelerator, combination thereof, or other such hardware structures.

[0073] The computing system 500 may include an input / output (I / O) interface 504. The I / O interface 504 may enable connection to input devices and / or output devices (not shown).

[0074] The computing system 500 may include a network interface 506 for wired or wireless communication with other computing systems (e.g., template database 150, user equipment 10, etc.). Network interface 506 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for intra-network and / or inter-network communication. Network interface 506 may also enable computing system 500 to transmit output (e.g., generated interpretations) to another computing system (e.g., to user equipment 10).

[0075] Computing system 500 may include storage unit 508.Storage unit 508 may include high-capacity storage units such as solid-state drives, hard disk drives, disk drives, and / or optical disk drives.

[0076] Computing system 500 may include memory 510. Memory 510 may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). Non-transient memory 510 may store instructions for execution by processing unit 502 to perform exemplary embodiments described herein, etc. For example, memory 510 may store instructions 512 for implementing the interpretable template retrieval system 100 and any methods disclosed herein. Memory 510 may also store neural network parameters. These neural network parameters may be parameters learned from training a neural network, such as the learning parameters of a neural network implemented in the interpretable template retrieval system 100. Memory 510 may include other software instructions to implement an operating system and other applications / functions, etc.

[0077] The computing system 500 may additionally or alternatively execute instructions in external memory (e.g., an external drive that communicates with the server via wired or wireless communication), or may be provided with executable instructions via transient or non-transient computer-readable media. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable storage.

[0078] Referring again to FIG1. ​​The interpretable template retrieval system 100 may be used as part of an optimization modeling system and / or with a solver to model and solve optimization problems. In an example discussed further below, an incremental modeling approach is described in which the optimization model can be constructed by retrieving templates and generating interpretations for each retrieved template to explain relevance. This improves the efficiency of the optimization model (and thus the solution of the optimization problem) generation process and reduces the need for human input (e.g., input by OR experts).

[0079] First, the details of the refined query generator 200 will be discussed.

[0080] Figure 3 is a block diagram of an exemplary implementation of the refined query generator 200. Although some functional blocks are shown in the refined query generator 200, it should be understood that this is not limiting.

[0081] The refined query generator 200 can access the structured model summary 202. The structured model summary 202 is a data structure that can be used to track and retrieve the basic model template and the components required to generate the optimized model.The structured model summary 202, as shown in Figure 3A, is implemented within the fine-grained query generator 200; however, in other examples, the structured model summary 202 may be external to the fine-grained query generator 200. For example, the structured model summary 202 may be a data structure stored throughout the interpretable template retrieval system 100. The structured model summary 202 may initially be empty. For example, the structured model summary 202 (see specification 10 / 27 pages 15 CN 121153030 A) may require information to identify the entities of the model, key performance indicators (KPIs) or optimization objectives, and any contextual rules (e.g., business rules and other constraints). The fine-grained query generator 200 may define initial queries (which may be predefined natural language questions) related to the information required to complete the structured model summary 202, such as, “What are the entities in the problem?”, “What key KPIs do you want to include?”, and “What are the business rules?”. In some examples, the user may input the initial query (which may be a natural language query) through user device 10.

[0082] The fine-grained query generator 200 may also include a segmentation module 204. The segmentation module 204 segments the context information (which may be provided by the user device 10 as a single long text document) into context fragments (e.g., sentences). It is anticipated that in many or most cases, the context information may be too long to be processed in its entirety. For example, the context information may be provided as a text document that is hundreds of pages long and contains thousands or tens of thousands of sentences. In these cases, the context information may be processed in portions at a time (e.g., using a moving window). Therefore, the segmentation module 204 may generate context fragments corresponding to a portion of the context information (rather than all of the context information).

[0083] The fine-grained query generator 200 uses a trained first NLP encoder 206 (e.g., a trained encoder-only converter, such as a BERT model). The first NLP encoder 206 has been trained to encode the context fragments and the initial query into a common vector space. Specifically, the first NLP encoder 206 has been trained (discussed in detail below) such that related context fragments and the initial query are encoded as corresponding context vectors and query vectors that are close to each other in the vector space. The first NLP encoder 206 encodes the initial query (e.g., a predefined initial query or an initial query input by the user) into a corresponding query vector. In some examples, if the initial query is predefined, the corresponding query vector can be generated in advance using the first NLP encoder 206 and stored by the fine-grained query generator 200, thus eliminating the need to generate the query vector every time.This may help reduce the required runtime computation and facilitate the real-time execution of the interpretable template retrieval system 100 (e.g., the user is able to interact with the interpretable template retrieval system 100 and receive output from the interpretable template retrieval system 100 with almost no latency). Furthermore, the first NLP encoder 206 is used to encode context fragments into context vectors.

[0084] The query vector and context vector are input to the similarity and connection module 208. The similarity and connection module 208 compares each query vector with each context vector to identify the most similar query-context vector pairs. It should be noted that since a context fragment may correspond to only a portion of the context information (e.g., in cases where the original context information is too long to be fully processed), the context fragment may or may not include the context associated with each initial query. Therefore, the similarity and connection module 208 may perform various operations to ensure that only query-context vector pairs that satisfy a minimum similarity threshold are identified.

[0085] For example, the similarity and connection module 208 can calculate a similarity metric (e.g., cosine similarity) between each query vector and each context vector (that is, if there are n query vectors and m context vectors, then n×m similarity metrics are calculated). Then, one or more query-context vector pairs whose similarity metrics exceed a predefined similarity threshold (e.g., at least 70% similarity) are identified as one or more matching pairs (wherein, the query vector and context vector in the matching pair may be referred to as mutually matching). Based on one or more matching pairs, the similarity and connection module 208 can select a query corresponding to the query vector with the most matching context vectors. Then, one or more context fragments corresponding to one or more context vectors that match the selected query are also selected. The similarity and connection module 208 connects the selected query with one or more selected context fragments to generate a connected query. The connected query may include tags identifying the query and one or more context fragments.

[0086] The connected query is input into a first NLP generative model 210 (e.g., an encoder-decoder converter model, such as a BART model). The first NLP generator model 210 is trained to generate a fine-grained natural language query, such as "11 / 27 pages, 16 CN 121153030 A", from the concatenated query. The fine-grained query can then be passed to the template retrieval unit 300.

[0087] In some examples, the process of generating a fine-grained query may include (e.g., presenting the fine-grained query to a user via a user interface (UI) on the user device 10) for the user to approve, reject, or modify. After user approval, the fine-grained query can be passed to the template retrieval unit 300.When a user can approve a detailed query, the presentation of the detailed query can be enhanced by presenting the initial query and predefined metadata explaining the initial query and the extracted contextual information, so that the user can understand the detailed query.

[0088] For example, part of the contextual information could be "I want to provide services to all customers, balance the workload of drivers to avoid overtime, and reduce overall operating costs." The initial query could be a predefined question related to the structured model summary 202, such as "What key metrics do you want to observe (maximize or minimize)?" The initial query can be obtained from a predefined list of general questions, which correspond to different requirements for completing the structured model summary 202. The purpose of designing these questions may be to prompt the user to verify information such as the objective, whether time is a dimension, the entity of the question, and the question type. It should be noted that even if the initial query is selected from the predefined list of questions, the presentation order of these questions may not be predetermined. Using the operations of the fine-grained query generator 200 described above, the generated fine-grained query could be "What key metrics do you want to observe (maximize or minimize)? Based on your description, your goal is 'to reduce overall operating costs.' Do you want to 'minimize operating costs defined by 'drivers' time on the road, daily number of employees, and overtime pay'?" In this example, the potential goal "to reduce overall operating costs" and the factors influencing "costs" are identified in the context information. Users can accept, revise, or reject the fine-grained query.

[0089] Figure 3B shows another example of a fine-grained query that can be generated using the fine-grained query generator 200.

[0090] In this example, only an initial query 222 is shown. The initial query 222 can be a predefined question designed to gather the information needed to complete the structured model summary 202. The initial query 222 (by the first NLP encoder 206) is encoded as a query vector 224. Three context fragments 226 (which can be segmented from a longer document including context information) are shown, each context fragment 226 (by the first NLP encoder 206) being encoded as a context vector 228. (For example, the similarity and connection module 208) identifies and connects the query-context vector pairs with the highest similarity (similar shaded representations in Figure 3B), and inputs these query-context vector pairs into the first NLP generator model 210 to generate a refined query 230. In this example, the refined query 230 is presented on user device 10 for user approval. The refined query 230 is presented along with additional information 232, such as the initial query, a general explanation of the initial query, and extracted contextual information. The user can approve, reject, or modify the refined query (e.g., edit the text). The approved refined query can then be passed to the template retrieval unit 300.

[0091] The fine-grained query generator 200 in the interpretable template retrieval system 100 performs various operations to extract contextual information found in the question specification and add this contextual information to the initial query to form a fine-grained query. It should be noted that the above-described NLP-based method can extract context related to the query to form a fine-grained query. This may be more advantageous than other methods such as concatenating the question specification with the initial query, because the question specification may be very long and / or may include irrelevant or ambiguous information that may hinder template retrieval. By using the fine-grained query generator 200 described above, only the context related to the initial query can be extracted, thereby generating a relatively concise fine-grained query that also includes information-rich context. By including information about the context, the fine-grained query may help improve the performance of the template retriever 300 and the interpreter generator 400. The fine-grained query may optionally be presented to the user for review, revision, and approval.

[0092] Figure 4A is a block diagram of an exemplary implementation of the template retriever 300. Although some functional blocks of the template retriever 300 are shown, it should be understood that this is not limiting.

[0093] Template retrieval 300 receives fine-grained queries (which may have been approved by the user). Template retrieval 300 may also access or include template database 150 (Figure 4A shows template database 150 outside template retrieval 300, but in some examples, template database 150 may be part of template retrieval 300). Templates stored in template database 150 are associated with metadata (e.g., semantic information such as a general interpretation of the template) and some symbolic content (e.g., mathematical content).

[0094] Fine-grained queries are encoded into fine-grained query vectors by a trained second NLP encoder 302 (e.g., a trained encoder-only converter, such as a BERT model). In addition, semantic metadata associated with each template is also encoded into template vectors by the second NLP encoder 302. The second NLP encoder 302 has been trained to encode fine-grained queries and template metadata into vectors in a common vector space. Specifically, the second NLP encoder 302 has been trained (discussed in detail below) so that related fine-grained query and template metadata are encoded into corresponding fine-grained query vectors and template vectors that are close to each other in the vector space. In some examples, the template metadata can be pre-encoded into corresponding template vectors using the second NLP encoder 302 and stored (e.g., along with the corresponding templates in the template database 150, or by the template retrieval unit 300), thus eliminating the need to generate template vectors every time.This may help reduce the required runtime computation and facilitate the real-time execution of the interpretable template retrieval system 100 (e.g., the user is able to interact with the interpretable template retrieval system 100 and receive output from the interpretable template retrieval system 100 with almost no latency).

[0095] The fine-grained query vector and template vector are input into the similarity module 304. The similarity module 304 calculates a similarity metric (e.g., cosine similarity) between the fine-grained query vector and each template vector. The similarity module 304 may identify one or more template vectors that are sufficiently similar to the query vector (e.g., the similarity metric exceeds a predefined threshold, e.g., at least 70%) and / or may identify the top few (e.g., the top three) most similar template vectors. One or more templates corresponding to the identified one or more template vectors may be retrieved from the template database 150. The one or more retrieved templates may then be passed to the interpretation generator 400. In some examples, template retrieval 300 may also pass a fine query (received by template retrieval 300 from fine query generator 200) to interpretation generator 400; in other examples, interpretation generator 400 may alternatively receive a fine query from fine query generator 200.

[0096] Consider a fine query as an example of “I want to minimize operating costs defined by “drivers’ on-the-road time, number of employees per day, and overtime pay”.” The second NLP encoder 302 may process this fine query as input and generate a corresponding fine query vector. If template database 150 stores template T1 with metadata “minimize total travel time”, template T2 with metadata “minimize driver costs”, and template T3 with metadata “maximize vehicle usage”, the second NLP encoder 302 may process the metadata of each of templates T1, T2, and T3 as input and generate a template vector. The similarity module 304 can calculate a similarity metric (e.g., cosine similarity) between the fine-grained query vector and each of these template vectors, and identify the template vector corresponding to template T2 as the most similar to the fine-grained query vector. Therefore, template T2 with metadata “minimize driver cost” (and additional mathematical content) is retrieved from the template database 150.

[0097] In some examples, more than one template can be retrieved. For example, if the fine-grained query is entity-related, the query may include multiple entities for which multiple templates need to be retrieved. Figure 4B illustrates an example of this situation.

[0098] A fine-grained query 322 is shown, which may have been generated by the fine-grained query generator 200 and passed to the template retrieval unit 300. The fine-grained query 322 (encoded by the second NLP encoder 302) is encoded into a fine-grained query vector 324.In this example, there may be N templates 326 stored in the template database 150, each template having metadata that provides a common meaning for each template 326. In this example, template T1 is a template for the entity "driver" and includes metadata defining the meaning of the template ("driver"), synonyms ("truck driver, car driver, bus driver, truck driver"), and related entities ("vehicle, customer"). Metadata may include tags indicating the type of information in the metadata. For example, the metadata of an entity template may include tags [NAME] for indicating the entity name, tags [VARIANTS] for indicating entity variants or synonyms, and / or tags [COMMON_ENTITIES] for indicating related entities. For example, the metadata of T1 can have the following form:

[0099] [CLS][NAME] Driver [VARIANTS] Truck driver, car driver, bus driver, truck driver [COMMON_ENTITIES] Vehicle, customer [SEP]

[0100] Wherein, the marker [CLS] indicates the beginning of the text in the metadata, and [SEP] indicates the end of the text. It should be noted that in some examples, the marker may not be needed; instead, the metadata can have a predefined format or predefined fields.

[0101] Template T2 is a template for the entity “package”, and template TN is a template for the entity “city”; each template 326 includes metadata of a similar type. The metadata of each template 326 (encoded by the second NLP encoder 302) is a template vector 328. (For example, the similarity module 304) finds that there are two template vectors 328 that have sufficient similarity to the fine-grained query vector 324 (as shown in the similar shaded representation in Figure 4B). Therefore, T1 and T2 are retrieved templates 330 retrieved from the template database 150. The retrieved template 330 can be passed to the interpretation generator 400.

[0102] FIG5A is a block diagram of an exemplary implementation of the interpretation generator 400. Although some functional blocks are shown in the interpretation generator 400, it should be understood that this is not limiting.

[0103] The interpretation generator 400 receives the fine query and the retrieved template. It should be noted that if the template retriever 300 retrieves multiple retrieved templates, the interpretation generator 400 can process the retrieved templates one by one to generate the corresponding generated interpretation.

[0104] The metadata of the fine query and the retrieved template can be concatenated together by the connection module 402 to form a concatenated string. The connection module 402 may optionally add tags to indicate the start and / or end of the query and the start and / or end of the template metadata within the concatenated string.

[0105] The concatenated string is input into the second NLP generation model 404 in the interpretation generator 400. The second NLP generation model 404 may be, for example, an encoder-decoder converter model (e.g., a BART model), which has been trained to generate natural language interpretations from the concatenated fine-grained query and template metadata.

[0106] For example, if the fine-grained query is “I want to minimize the operating costs defined by “drivers’ on-the-road time, number of employees per day and overtime pay”” and the template metadata is “minimize the total travel time”, then the concatenated string may be “[QUERY] I want to minimize the operating costs defined by “drivers’ on-the-road time, number of employees per day and overtime pay” “[TEMPLATE] minimize the total travel time”. This can be processed by the second NLP generator mode 404 to generate the above-mentioned generated explanation, “You mentioned ‘minimizing operating costs,’ and costs depend on ‘driver’s driving time.’ Therefore, by minimizing the total driving time, costs can be minimized.” Therefore, the explanation generator 400 provides a generated explanation using contextual information included in the granular query and similarities found in the template metadata. The generated explanation uses a language that is appropriate to the specific context of the problem (e.g., including skill level, user style, problem domain, etc.) to explain the template.

[0107] The generated explanation can then be output to be presented to the user via user device 10 (e.g., via a UI, as discussed further below). The generated explanation can be displayed together with the retrieved template and / or granular query. In some examples, the metadata in the retrieved template (if applicable, after user approval) can be appended to the context information to update the context information. The updated context information can be used to generate subsequent granular queries.

[0108] If multiple retrieved templates exist, the explanation generator 400 can generate multiple generated explanations, and each generated explanation can be presented together with the corresponding retrieved template. Figure 5B illustrates an example where a fine-grained query has multiple generated interpretations.

[0109] In this example, a fine-grained query 422 and metadata 424a, 424b from two templates are provided to an interpretation generator 400. The fine-grained query 422 is concatenated with the first metadata 424a, and then (e.g., a second NLP generation model 404) the fine-grained query 422 is processed to generate a first generated interpretation 426a. Next, the fine-grained query 422 is concatenated with the second metadata 424b, and both are processed as described on page 14 / 27 of the specification, CN 121153030 A, to generate a second generated interpretation 426b.

[0110] The generated interpretations 426a and 426b can both be presented together with the corresponding retrieved templates for user approval, rejection, or modification, etc.If the user approves the template, the structured model summary can be populated with the data of the approved template (e.g., symbolic mathematical content and optional metadata).

[0111] The following describes some examples of how to train NLP models (e.g., first NLP encoder 206, first NLP generative model 210, second NLP encoder 302 and second NLP generative model 404) in the interpretable template retrieval system 100.

[0112] In some examples, both the first NLP encoder 206 and the second NLP encoder 302 can be encoder-only converters, such as BERT models. The encoder-only converter can be fine-tuned using data collected from multiple domains (e.g., corresponding to different business domains that are typically associated with optimization problems, such as distributions).

[0113] The first NLP encoder 206 can be fine-tuned using a training dataset that pairs context fragments (e.g., sentences in a natural language context description) with natural language queries that match model components. Both the context fragments and the paired queries are processed by the first NLP encoder 206 to generate context vectors and query vectors. A similarity metric (e.g., cosine similarity) between the two vectors is then computed. The purpose of training can be to maximize the similarity between two vectors. For example, the computed similarity metric can be used (e.g., using backpropagation) to update the parameters of the task-specific layer of the first NLP encoder 206.

[0114] The second NLP encoder 302 can be fine-tuned using a training dataset that pairs fine-grained natural language queries with labeled metadata of model component templates that satisfy the fine-grained queries. For example, the fine-grained query “We have 5 trucks and 10 drivers. The working hours for employees and drivers are Monday to Saturday, 6 a.m. to 6 p.m.” can be paired with labeled metadata of model entity templates that are associated with vehicles such as: [CLS][NAME] vehicles [VARIANTS] trucks, vans, cars, SUVs, four-wheeled vans, handcarts, ships, boats, trains, railways, freight cars, taxis, airplanes, bicycles, scooters, electric bicycles [COMMON_ENTITIES] drivers, warehouses, packages, customers [SEP]. The training dataset can include data pairs for different types of model components (e.g., including different entities, different decisions, different objectives, different constraints, etc.) and for different domains. Both the fine-grained query and the paired metadata are processed by the second NLP encoder 302 to generate fine-grained query vectors and template vectors. A similarity metric (e.g., cosine similarity) between the two vectors is then computed. The goal of training can be to maximize the similarity between the two vectors. For example, the computed similarity metric can be used (e.g., using backpropagation) to update the parameters of a task-specific layer of the second NLP encoder 302.

[0115] In some examples, both the first NLP generative model 210 and the second NLP generative model 404 can be encoder-decoder converters, such as the BART model. The encoder-decoder converter can be trained to perform a sequence-to-sequence text generation task. The encoder-decoder converter can be fine-tuned to generate refined queries or interpretations.

[0116] For example, the first NLP generative model 210 can be fine-tuned using a training dataset comprising data triples, where each triple includes an initial query, relevant context fragments, and a refined query to be generated. The refined query in each data triple can be created manually or semi-automatically (e.g., roughly generated by software and further edited by a human).

[0117] Similarly, the second NLP generative model 210 can be fine-tuned using a training dataset of data triples, where each triple includes template metadata, relevant refined queries, and a context-based interpretation to be generated. The context-based interpretation in each data triple can be created manually or semi-automatically.

[0118] As described above, the interpretable template retrieval system 100 performs various operations to provide retrieved templates and context-based explanations. This provides information that allows a user to select or approve a template, which can then be used to generate an optimized model. Including context in a granular query also facilitates the retrieval of relevant templates. Since a granular query includes more information than a general query, a more suitable or appropriate template can be retrieved. Since templates are used as building blocks for generating an optimized model, improving template retrieval may also contribute to improving the generation of the optimized model.

[0119] The generated explanation takes into account style information (e.g., level of English comprehension, use of technical language specific to the application domain, etc.) and the problem context and / or the level of OR expertise indicated in the contextual information, thereby helping to ensure that the user can understand the generated explanation.

[0120] The interpretable template retrieval system 100 allows users to better understand and control the generation process of the optimized model without requiring the user to be an OR expert. The interpretable template retrieval system 100 can be used at different stages of generating the optimized model and for retrieving different templates. For example, the interpretable template retrieval system 100 can be used to retrieve individual components (e.g., constraints, objectives, metrics, entities, etc.) in the entire model, the base model, and / or the optimization model. An example of how the interpretable template retrieval system 100 can be used in an incremental modeling approach to generate an optimization model is described below.

[0121] The interpretable template retrieval system 100 can be used at different stages of the incremental modeling approach to retrieve templates and provide an explanation of the relevance of the template to the optimization problem.This process can be performed without human OR experts and can enable non-OR users to generate optimized models without any coding knowledge.

[0122] Figure 6 is a flowchart of an exemplary incremental modeling method 600 that can be performed by a computing system (e.g., as shown in Figure 2) that implements the above-disclosed interpretable template retrieval system 100. In some examples, the incremental modeling method 600 can be implemented by other computing systems and is not necessarily limited to the interpretable template retrieval system 100. The incremental modeling method 600 can be performed by a computing system that implements an incremental modeling system (which may be based on or include the interpretable template retrieval system 100, or may not be based on or include the interpretable template retrieval system 100).

[0123] It should be noted that user interaction with the interpretable template retrieval system 100 can be supported in each or any step of the incremental modeling method 600. As discussed further below, in some examples, user interaction (e.g., receiving user input and / or providing user output) can be performed through a UI provided by a computing system (e.g., a computing system that also implements the interpretable template retrieval system 100).

[0124] In 602, context information is received. For example, context information can be received from a user via user equipment 10 communicating with a computing system (e.g., via a wireless network). The context information includes at least a textual description of the optimization problem to be modeled, which can be described using natural language. The context information can be provided to the system by the user uploading a text document including the problem specification. The context information can be any natural language text, with no requirements on structure or content. The context information can describe the context of the optimization problem with any level of detail and any level of technology based on the user's knowledge, etc.

[0125] In 604, a structured model summary can be initiated. The structured model summary can be a data structure that includes some default data fields. The structured model summary can provide a structure for storing information about the model components (e.g., entities, one or more constraints, one or more objectives, etc.) required to properly describe the optimization problem. In some examples, this step can be omitted.

[0126] In 606, the structured model summary is updated using one or more retrieved templates, which can be retrieved from a template database (e.g., template database 150). Each template is associated with a model component. Each template can be retrieved using a granular query, which is generated from and includes contextual information (which may help in retrieving relevant templates). The granular query can be a natural language query, including natural language text from the contextual information, as well as information related to the model components to be retrieved. The granular query can be presented to the user (e.g., via user device 10) for review, editing, and / or approval before the template is retrieved using the granular query.In some examples, a fine-grained query may be presented to the user only when it relates to complex model components such as decisions and constraints, and may not be presented when it relates to simple model components such as Entity Specification Pages 16 / 27 21 CN 121153030 A. In some examples, whether to present a fine-grained query to the user or what type of fine-grained query to present for review, editing and / or approval may be a user-configurable option. In some examples, metadata from the retrieved templates may also be used to update contextual information. For each retrieved template, a context-based explanation may be generated and presented to the user so that the user can review and approve (or reject) the retrieved templates for use by the structured model summary.

[0127] Step 606 may be performed iteratively (as indicated by the curved arrow), so that the structured model summary is updated incrementally (e.g., by retrieving templates for different model components until all required fields of the structured model summary are updated with information from the retrieved templates).

[0128] For example, the iteration of step 606 may include (e.g., obtaining an initial query optimized into a fine-grained query from user input or from a predefined list of queries associated with the structured model summary). Optionally, the fine-grained query may be presented for user review and approval. The fine-grained query may then be used to retrieve a model specification template (e.g., a template for a model entity or model target). A context-based explanation may be generated to associate the retrieved template with context information and presented to the user for approval. After the template is retrieved (optionally approved by the user), the metadata associated with the template (which may be a natural language description of the template) may be used to update the context information (e.g., the template's metadata may be appended to the context information), and the template may be used to update the structured model summary. In some examples, in addition to using the template metadata to update the context information, the generated context-based explanation may be used alternatively or additionally to update the context information.

[0129] In 608, the base model template is retrieved using the structured model summary (which has been updated in step 606) as the query. The structured model summary can be used to query a template database (e.g., template database 150) that stores the base model template. A symbolic base model associated with the base model template is also retrieved. The symbolic base model can serve as the basis for constructing a final symbolic model, which will become the optimized model. The symbolic model includes context-independent symbolic content that may be difficult for non-OR experts to understand. Therefore, step 608 may include generating a context-based explanation to associate the base model template with the structured model summary. The context-based explanation may be presented to the user (e.g., via user device 10) for review and / or approval.In some examples, if multiple base model templates are retrieved (e.g., multiple base model templates are found to match the structured model summary), a context-based explanation of all retrieved base model templates can be presented, and a user-selected base model template can be received.

[0130] In 610, an optimization model is constructed based on the base model templates using one or more retrieved templates. That is, the base model templates can be used as a starting point for the optimization model, and one or more additional retrieved templates can be used to progressively build the optimization model. For example, constraints and / or objectives can be added to build the optimization model from the starting point (i.e., the base model templates) into a more complex optimization model to better represent the optimization problem. (e.g., from template database 150) One or more templates of one or more additional model components to be added to the optimization model are retrieved, which are used to update the symbolic model associated with the optimization model. Each template of an additional model component can be retrieved using a fine-grained query. For example, the retrieved base model templates can predefine additional templates that can be added to the optimization model. Thus, the initial query (related to the predefined templates) can be predefined based on the base model templates, and the fine-grained query can be generated based on the initial query. For example, if the base model template does not include the template included in the structured model summary, the metadata of the omitted template can be used as a fine-grained query to add the template to the optimized model. For another example, the fine-grained query can be generated based on an initial query input by the user. Similar to step 606, the fine-grained query can be presented to the user (e.g., via user device 10) for review, editing, and / or approval before retrieving the template using the fine-grained query. Optionally, each retrieved template and a context-based explanation can be provided to the user to enable the user to approve the retrieved template. Step 610 can be performed iteratively (as indicated by the curved arrow), so that the symbolic model is progressively updated (e.g., by retrieving templates for different symbolic components to add to the symbolic model (optionally approved by the user), until the final symbolic model is generated).

[0131] In 612, the final optimized model and associated symbolic model generated from one or more iterations of step 610 are output as the generated optimized model. The optimization model can be stored for future use and / or the optimization model can be instantiated into a model instance using data values, and then the solver solves the model instance.

[0132] It should be understood that although the incremental modeling method 600 is described as a method or process, the incremental modeling method 600 disclosed herein can be embodied as a system. For example, a computer-implemented incremental modeling system (which may be an embodiment of the computing system shown in FIG2) can execute instructions to implement the incremental modeling method 600.Furthermore, one or more steps of the incremental modeling method 600 can be executed by executing instructions from subsystems or software modules in the incremental modeling system. For example, step 606 can represent the operation of the structured model summary clarifyer subsystem in the incremental modeling system; step 608 can represent the operation of the basic model template retriever and interpreter subsystem in the incremental modeling system; and step 610 can represent the operation of the model builder subsystem in the incremental modeling system.

[0133] Furthermore, one or more steps of the incremental modeling method 600 can be executed by the interpretable template retrieval system 100. For example, in step 606, the interpretable template retrieval system 100 can be used to retrieve templates that update structured model summaries and context information; in step 608, the interpretable template retrieval system 100 can be used to retrieve basic model templates and generate template interpretations; and in step 610, the interpretable template retrieval system 100 can be used to retrieve templates for building additional model components for optimizing the model.

[0134] Figure 7 illustrates the details of exemplary steps for executing step 606. The steps shown in Figure 7 can be executed by the interpretable template retrieval system 100, as discussed further below.

[0135] Prior to step 606, contextual information is received and a structured model summary is initiated. Step 606 can be performed via steps 710, 730, and 750, each of which can include sub-steps as described below.

[0136] In 710, a refined query is generated. This refined query includes relevant contextual information that can be extracted from the contextual information using a trained NLP model. Step 710 can be performed by the refined query generator 200. In some examples, performing step 710 may include performing steps 712 through 720.

[0137] In 712, a query vector is obtained. This query vector is obtained by encoding an initial query (which can be received from user input or from a predefined query list via user device 10). The initial query is encoded into a query vector using a trained NLP encoder (e.g., using the first NLP encoder 206 in the refined query generator 200). If the initial query comes from a predefined query list, the initial query can be presented to the user for review and / or approval. If the initial query comes from a predefined list of queries, a query vector may have already been generated using a trained NLP encoder and stored in memory before incremental modeling method 600; then, the query vector can be retrieved in step 712 by retrieving it from memory. In this example, there is one initial query and one corresponding query vector. In other examples, there may be multiple initial queries (and multiple corresponding query vectors).

[0138] In 714, the context information is segmented into context fragments (e.g., using the segmentation module 204 in the fine-grained query generator 200), and the context fragments are encoded into context vectors using a trained NLP encoder (e.g., using the first NLP encoder 206 in the fine-grained query generator 200).

[0139] In 716, one or more context fragments corresponding to one or more context vectors similar to the query vector are selected based on the similarity between the query vector and each context vector. For example, the similarity and connection module 208 in the fine-grained query generator 200 can be used to compute a similarity metric (e.g., cosine similarity) and identify one or more context vectors sufficiently similar to the query vector. Then, one or more context fragments corresponding to the identified one or more context vectors can be selected. Specification 18 / 27 pages 23 CN 121153030 A

[0140] In 718, a fine-grained query is generated. For example, one or more selected context fragments can be combined with the initial query (e.g., connected). The combined string can be provided as input to a trained NLP generative model (e.g., the first NLP generative model 210 in the fine-grained query generator 200) to obtain a fine-grained query. In this example, one fine-grained query is generated. In other examples, multiple fine-grained queries can be generated (e.g., corresponding to multiple selected context fragments).

[0141] Optionally, in 720, a fine-grained query approved by the user can be received. For example, the generated fine-grained query can be presented to the user (e.g., via a UI on user device 10) for acceptance, rejection, or modification.

[0142] In 730, one or more templates are retrieved. One or more templates can be retrieved using the fine-grained query generated in step 720. A trained NLP model can be used to perform template retrieval. Step 730 can be performed by the template retrieval unit 300. Performing step 730 can include performing steps 732 to 736, etc.

[0143] In 732, a refined query (optionally after user approval and / or modification) is encoded into a refined query vector using a trained NLP encoder (e.g., using the second NLP encoder 302 in template retrieval 300).

[0144] Optionally, in 734, template metadata associated with a template stored in a template database can be encoded into a template vector using a trained NLP encoder (e.g., using the second NLP encoder 302 in template retrieval 300). In some examples, the template metadata may have already been encoded into a template vector prior to incremental modeling method 600, and the template vector may have been stored in memory (e.g., in template database 150); in this case, step 734 can be omitted, and the previously generated template vector can be retrieved from memory.

[0145] In 736, based on the similarity between the fine-grained query vector and each template vector (whether retrieved from memory or by performing step 734), one or more template vectors corresponding to one or more template vectors similar to the fine-grained query vector are retrieved. For example, the similarity module 304 in the template retriever 300 can be used to compute a similarity metric (e.g., cosine similarity) and identify one or more template vectors that are sufficiently similar to the fine-grained query vector. For example, one or more template vectors with a similarity metric exceeding the minimum similarity (e.g., at least 70% similarity) and / or one or more template vectors with the top few (e.g., top three) similarity metrics can be identified. Then, one or more templates corresponding to the identified one or more template context vectors can be retrieved from the template database. If multiple templates are retrieved, each retrieved template can be associated with a similarity metric computed for the corresponding template vector. The similarity metric can be used in subsequent steps to rank the retrieved templates.

[0146] In 750, a context-based explanation is generated for each retrieved template. One or more context-based explanations can be generated using a fine-grained query (generated in step 710) as a source of relevant context. A trained NLP model can be used to generate one or more explanations. Step 750 can be performed by explanation generator 400. Performing step 750 may include performing steps 752 and 754, etc.

[0147] In 752, for each retrieved template, the associated template metadata is combined with the fine-grained query to obtain a combined string. For example, the template metadata can be joined with the fine-grained query (e.g., using the connection module 402 in explanation generator 400). If there are multiple retrieved templates, this step is performed to obtain a combined string for each corresponding retrieved template.

[0148] In 754, an explanation is generated for each retrieved template using the corresponding combined string. For example, each combined string can be input into a trained NLP generation model (e.g., a second NLP generation model 404 in explanation generator 400) to obtain a corresponding generated explanation. The generated interpretation can be presented along with the corresponding retrieved templates for the user (e.g., via the UI on user device 10) to review, approve, reject, and / or modify.

[0149] Optionally, in 762, one or more of the retrieved templates selected by the user can be received. For example, the user can (e.g., via the UI on user device 10) select one or more templates they wish to use for optimizing the model.

[0150] In some examples, retrieved templates can be sorted according to an associated similarity metric, and the top-ranked templates (e.g., the top 3 templates with the highest similarity metric) can be automatically or by default selected (the user can further approve these top-ranked templates), instead of receiving one or more retrieved templates selected by the user.

[0151] In 764, the structured model summary and context information are updated using one or more selected templates. For example, if the initial query corresponds to a specific field in the structured model summary, that specific field can be populated using metadata and any symbolic content included in the selected templates. Any metadata associated with one or more retrieved templates can also be used to update context information (e.g., appended to a document that includes context information). Metadata can be simply appended to the end of the document or appended to relevant sentences detected as having the highest relevance and used to generate explanations.

[0152] Step 606 includes steps 710, 730, and 750 and their respective sub-steps. Step 606 can be repeated until all required fields in the structured model summary have been completed, or until an instruction (e.g., user input) to continue retrieving the base model template is received in step 608.

[0153] Figure 8 illustrates details of exemplary steps for performing step 608. The steps shown in Figure 8 can be performed by the interpretable template retrieval system 100, as discussed further below.

[0154] Prior to step 608, the structured model summary has been updated, and the context information has also been updated (e.g., in step 606 described above). Step 608 can be performed via steps 830 and 850, each of which can include sub-steps as described below.

[0155] In 830, the updated structured model summary is used as a query to the template database 150 to retrieve one or more candidate templates. A trained NLP model can be used to perform the template retrieval. In step 830, the updated structured model summary may have functionality similar to the fine-grained query in step 730 discussed earlier, i.e., the updated structured model summary includes information on relevant candidates for retrieving the base model template. Step 830 may be performed by the template retrieval unit 300. Performing step 830 may include performing steps 832 to 836, etc.

[0156] In 832, the updated structured model summary is encoded into a fine-grained query vector (also referred to as a model summary vector) using a trained NLP encoder (e.g., using the second NLP encoder 302 in the template retrieval unit 300).

[0157] Optionally, in 834, template metadata associated with candidate base model templates stored in the template database is encoded into template vectors using a trained NLP encoder (e.g., using the second NLP encoder 302 in template retrieval 300). In some examples, the template metadata may have already been encoded into template vectors before incremental modeling method 600, and the template vectors may be stored in memory (e.g., in template database 150); in this case, step 834 can be omitted, and the previously generated template vectors can be retrieved from memory.

[0158] In 836, one or more candidate templates corresponding to one or more template vectors similar to the fine-grained query vector are retrieved based on the similarity between the fine-grained query vector and each template vector (whether retrieved from memory or by performing step 834). This can be similar to step 736 discussed above and will not be repeated here.

[0159] In 850, a context-based explanation is generated for each candidate template. One or more context-based explanations can be generated using updated context information as a source of relevant context. A trained NLP model can be used to generate one or more explanations. Step 850 can be performed by the interpreter generator 400. Performing step 850 may include performing steps 852 and 854, etc.

[0160] In 852, for each candidate template, the template metadata is enhanced by contextual enhancement to obtain enhanced template metadata. For example, if the updated context information is relatively short (e.g., no more than a paragraph or 500 words), the context information can be concatenated with the template metadata to obtain enhanced template metadata. As another example, the context may be included in the metadata of the structured model summary on pages 20 / 27 of the specification CN 121153030 A, and the metadata of the structured model summary may be concatenated with the metadata of the candidate template to obtain enhanced template metadata. Alternatively or additionally, the metadata associated with each model component in the candidate template may be paired with the metadata of the corresponding model component in the structured model summary (e.g., the metadata of an entity in the candidate template may be paired with the metadata of the corresponding entity in the structured model summary). For example, relevant context fragments or fine-grained queries for retrieving a specific entity in the structured model summary may be stored as metadata of the structured model summary. This metadata can then be paired with the corresponding entity in the candidate template. Therefore, the enhanced template metadata can be a collection of paired model component metadata. Generally, it should be understood that there are multiple ways to obtain the enhanced template metadata, and the examples described above are not limiting. If multiple candidate templates exist, this step is performed to obtain the enhanced template metadata for each corresponding candidate template.

[0161] In step 854, an explanation is generated for each candidate template using the corresponding enhanced template metadata. For example, each enhanced template metadata can be input into a trained NLP generative model (e.g., the second NLP generative model 404 in explanation generator 400) to obtain a corresponding generated explanation. The generated explanation can be presented along with the corresponding retrieved candidate template for review, approval, rejection, and / or modification by a user (e.g., via the UI on user device 10).

[0162] In step 862, one of one or more candidate templates selected by the user can be received. For example, a user can (e.g., via the UI on user device 10) select a base model template they wish to use as the basis for optimizing the model. In some examples, an automatically selected base model template approved by the user can be received instead of a user-selected base model template. For example, retrieved candidate templates can be sorted according to an associated similarity metric, and the top-ranked template (e.g., the candidate template with the highest similarity metric) can be automatically or by default selected (which the user can approve). The selected candidate template is used as the base model template for building the optimized model.

[0163] The base model template has a basic symbolic model and template metadata (which may include metadata associated with the entire base model template and metadata associated with each model component in the base model template). The base model template can be used as the initial data structure for the optimized model (e.g., similar to a structured model summary) and can replace the structured model summary to store records of selected model components. The symbolic model (and associated model code) can be stored in fields of the optimized model's data structure, while other fields can be used to store information about the model components. The optimized model can be constructed (e.g., using step 610) by retrieving templates of the model components to be added to the optimized model and updating the symbolic model with the symbolic content (e.g., model code) of each retrieved template. The structured model summary can be saved as a basis for determining which templates should be added to the optimized model (e.g., the base model template used as the initial optimized model may initially omit templates found in the structured model summary), and so on. The structured model summary can also provide context that can be used to generate interpretations when constructing the optimized model. It should be noted that although the optimized model can use the base model template as a starting point, the final optimized model may differ from the base model template.

[0164] After step 608, the retrieved basic model template can be used in step 610 to construct an optimized model by using the basic model template as the initial optimized model and adding one or more retrieved templates to one or more additional model components.Each time a model component is added to the optimized model, the optimized model is updated incrementally (e.g., by adding the template and metadata of the additional model component to the data structure of the optimized model, and adding the code corresponding to the additional model component to the symbolic model).

[0165] Figure 9 shows details of an exemplary step for performing step 610. The steps shown in Figure 9 can be performed by an interpretable template retrieval system 100, as discussed further below.

[0166] Prior to step 610, a base model template has been retrieved and selected as the starting point to generate the optimized model (e.g., in step 608 above), and updated context information has been generated (e.g., in step 606 of the above step specification, pages 21 / 27, CN 121153030 A). The optimized model can then be built by adding model components (e.g., adding components incrementally or iteratively one by one) until the desired optimized model is obtained. Step 610 can be performed by steps 910, 930, and 950, each of which can include sub-steps as described below.

[0167] In 910, a fine-grained query is generated to query additional model components that can be used to build the optimized model. The fine-grained query includes information about the relevant context, which can be extracted from the updated context information using a trained NLP model. Step 910 can be performed by the fine-grained query generator 200. In some examples, performing step 910 may include performing steps 912 through 920. Step 910 may be similar to step 710 described above.

[0168] In 912, a query vector is obtained. This query vector is obtained by encoding an initial query (which may be received from user input or from a predefined query list via user device 10).

[0169] For example, the initial query may be a predefined query based on model components, etc., which can be added to build the optimized model according to a base model template. Thus, the initial query may be a predefined query of a specific model component added to the optimized model. For example, the base model template may include some model components, but may also define one or more additional model components that can be added to build the optimized model. For example, the base model template may include an objective (e.g., a minimization function or a maximization function), but may not include any constraints. The initial query can be a predefined query about the constraints to be added to the model. If the initial query comes from a predefined list of queries, it can be presented to the user for review and / or approval. Alternatively, the initial query can be received from user input via user device 10. The initial query is encoded into a query vector using a trained NLP encoder (e.g., the first NLP encoder 206 in the fine-grained query generator 200).If the initial query comes from a predefined query list, a query vector may have already been generated using a trained NLP encoder and stored in memory before incremental modeling method 600; then, the query vector can be retrieved in step 912 by retrieving the query vector from memory.

[0170] In 914, the updated context information is segmented into model description fragments (e.g., using segmentation module 204 in fine query generator 200), and the model description fragments are encoded into model description vectors using a trained NLP encoder (e.g., using first NLP encoder 206 in fine query generator 200). In some examples, a structured model summary can be used to provide context. Fields in the structured model summary can be fragments in the structured model summary. For example, segmentation model 204 can format or otherwise process fields in the structured model summary into fragments, which can be encoded into model description vectors by a trained NLP encoder.

[0171] In 916, one or more model description fragments corresponding to one or more model description vectors similar to the query vector are selected based on the similarity between the query vector and each model description vector. This can be performed similarly to step 716 above.

[0172] In 918, a fine-grained query is generated. For example, one or more selected model description fragments can be combined (e.g., concatenated) with the initial query. The combined string can be provided as input to a trained NLP generative model (e.g., the first NLP generative model 210 in the fine-grained query generator 200), and then the fine-grained query is obtained. In this example, one fine-grained query is generated. In other examples, multiple fine-grained queries can be generated (e.g., corresponding to multiple selected model description fragments).

[0173] In some examples, the fine-grained query can be metadata associated with a template of a model component currently omitted from the optimized model in the structured model summary, rather than being generated from the initial query.

[0174] Optionally, in 920, a fine-grained query approved by the user can be received. For example, the generated fine-grained query can be presented to the user (e.g., via a UI on user device 10) for acceptance, rejection, or modification.

[0175] In 930, one or more templates are retrieved. One or more templates can be retrieved using the refined query generated in step 920. A trained NLP model can be used to perform template retrieval. Step 930 can be performed by the template retrieval unit 300. Performing step 930 may include performing steps 932 through 936, etc.

[0176] In 932, a refined query (optionally after user approval and / or modification) is encoded into a refined query vector using a trained NLP encoder (e.g., using the second NLP encoder 302 in template retrieval 300).

[0177] In 934, template metadata associated with templates stored in a template database is encoded into template vectors using a trained NLP encoder (e.g., using the second NLP encoder 302 in template retrieval 300).

[0178] In 936, one or more templates corresponding to one or more template vectors similar to the refined query vector are retrieved based on the similarity between the refined query vector and each template vector. The retrieved templates include metadata and symbolic content that provides symbolic representations of model components. For example, a template for an object may include the metadata “minimize driving time” and the symbolic content {TYPE:objective,SENSE:minimize,VAR:x[i],PARAM:none}, representing an objective with a target meaning of minimization and 1-D decision variables.

[0179] In step 950, a context-based explanation is generated for each retrieved template. One or more context-based explanations can be generated using a fine-grained query (generated in step 910) as a source of relevant context. A trained NLP model can be used to generate one or more explanations. Step 950 can be performed by explanation generator 400. Performing step 950 may include performing steps 952 and 954, etc.

[0180] In step 952, for each retrieved template, the associated template metadata is enhanced by a fine-grained query. For example, the template metadata can be concatenated with the fine-grained query (e.g., using connection module 402 in explanation generator 400). If multiple retrieved templates exist, this step is performed to obtain a combined string of each corresponding retrieved template.

[0181] In step 954, an explanation is generated for each retrieved template using the corresponding enhanced template metadata. For example, each enhanced template metadata can be input into a trained NLP generation model (e.g., a second NLP generation model 404 in explanation generator 400) to obtain a corresponding generated explanation. The generated interpretation can be presented along with the corresponding retrieved templates for a user (e.g., via the UI on user device 10) to review, approve, reject, and / or modify.

[0182] Optionally, in 962, one or more templates from one or more retrieved templates selected by the user can be received. For example, a user (e.g., via the UI on user device 10) can select one or more templates for one or more model components that they wish to add to the optimized model.

[0183] In some examples, retrieved templates can be sorted according to an associated similarity metric, and the top-ranked templates (e.g., the top 3 templates with the highest similarity metric) can be automatically or by default selected (the user can further approve these top-ranked templates), instead of receiving one or more retrieved templates selected by the user.

[0184] In step 964, one or more selected templates are added to the optimization model and the symbolic model. For example, if the selected template is a template for a specific model component, the metadata of that model component can be added to the optimization model, and the symbolic content of that model component can be added to the symbolic model.

[0185] Step 610 includes steps 910, 930, and 950 and their respective sub-steps, and step 610 can be repeated to continue building the optimization model until user input is received to approve the optimization model. For example, after each iteration of step 610, the current state of the optimization model can be checked to determine if there are any model components that may need to be added and / or the user can be prompted to approve or continue building the model (e.g., via the UI on user device 10). For example, model components in the current optimization model can be compared with a predefined list of commonly used or expected model components associated with the base model template, and any model components not in the predefined list that are not in the current optimization model can be identified as model components that can be added to the optimization model. Similarly, the current optimization model can be compared with a structured model summary, and any model components found in the structured model summary but not in the current optimization model can be identified as model components that can be added to the optimization model (see page 23 / 27, 28 CN 121153030 A). Furthermore, the user can be prompted to provide information that can be used to identify model components to be added to the optimization model. Once the user has fully built and approved the optimization model, it can be output (e.g., saved and / or instantiated). The optimization model can be output along with the final updated context information (i.e., context information already attached with information from the template metadata). The updated context information can provide information linking model components to specific context fragments, which can help the user understand the generated optimization pattern. The updated context information can also serve as a record of how the optimization model was built and as documentation of the optimization model.

[0186] Various examples of the present invention enable non-OR experts to generate optimization models from unstructured natural language descriptions of optimization problems. This may help improve the accessibility of existing solvers by enabling non-OR experts to generate optimization models and then instantiate these optimization models as model instances that the solver needs to solve.

[0187] In some examples, a UI may be provided to enable users to interact with the interpretable template retrieval system 100 and the incremental modeling method 600, thereby providing user input and viewing generated explanations, etc. The UI may be part of a modeling assistance service accessible by user device 10 via a wireless network, etc., wherein the interpretable template retrieval system 100 and the incremental modeling method 600 are provided to the user as a cloud-based service or an online service.

[0188] Examples of UIs disclosed herein may provide users with natural language prompts to provide contextual information (e.g., a textual description of the optimization problem, business domain, and any other relevant information) by uploading a text document or entering it into a text editing field, etc. Exemplary UIs may provide users with prompts to review, edit, and / or approve granular queries of model components, and may provide users with prompts with context-based explanations to review and / or approve retrieved templates. Exemplary UIs may provide user options to modify the optimization model, e.g., add constraints and objectives. Exemplary UIs may also enable users to provide natural language input describing the model components to be added to the optimization model.

[0189] Some examples of the UI are described below, but these examples are not limiting.

[0190] Figure 10A shows a first screen of an exemplary UI 1000, in which the user is prompted to provide contextual information for an optimization problem. UI 1000 may include natural language prompts 1002 for the user and may also include a text editing field 1004. Contextual information may be entered into the text editing field 1004. In some examples, UI 1000 may additionally or alternatively provide the option for the user to upload a text document including contextual information.

[0191] Figure 10B shows a second screen of an exemplary UI 1000, which may be presented in the structured model summary clarification phase (e.g., in step 606 described above). UI 1000 includes a visual representation of the structured model summary 1006 (which has been initialized in this example and does not yet include any templates of any model components, such as fields “Entity,” “Decision,” “Objective,” and “Business Rule” being blank). UI 1000 also displays a fine-grained query 1008. In this example, the fine-grained query 1008 includes a general initial query 1008a with a predefined general definition of the initial query (“What entity best describes the problem?”), and is connected to the context 1008b extracted from the context information. UI 1000 also provides an optional option 1010 for the user to select a template to add to the structured model summary. The optional option 1010 can be generated based on the metadata of the retrieved template.

[0192] Figure 10C shows an exemplary pop-up window 1012 of UI 1000.The exemplary pop-up window 1012 can provide a context-based explanation 1014 for a specific retrieved template of a model component (in this example, a model entity related to "vehicle"), etc. The model component can be used to populate the structured model summary data structure. In this example, the pop-up window 1012 also provides predefined text related to the template (e.g., metadata associated with the template). In this example, the text is a definition of the template that can be added to the structured model summary. For example, the pop-up window 1012 can be displayed when the user selects the corresponding selectable option 1010 of UI 1000.

[0193] Figure 10D shows a third screen of the exemplary UI 1000. In this example, the structured model summary 1006 has been populated with templates (e.g., templates for entities "vehicle", "driver", "warehouse", and "package") retrieved from the specification pages 24 / 27, 29 CN 121153030 A. UI 1000 displays an initial query 1014 and relevant context 1016 for generating a fine-grained query 1018 to retrieve a decision template. Detailed query 1018 is displayed for user review, editing, and / or approval. In this example, detailed query 1018 is displayed in text editing field 1020, allowing the user to directly edit detailed query 1018. An optional option 1022 is also provided to allow the user to approve detailed query 1018 (regardless of whether the user edits it). Another optional option 1024 can be provided, allowing the user to skip the generated detailed query 1018 (e.g., if the user wishes to consider a different query first).

[0194] When option 1022 for approving a detailed query (regardless of whether the user edits it) is selected, the interpretable template retrieval system 100 can use the detailed query to retrieve relevant templates. For example, a context-based explanation of the retrieved template can be displayed in a pop-up window similar to the exemplary pop-up window 1012 shown in FIG. 10C for user review. The user can select to approve the retrieved template, which can then be used to populate the structured model summary 1006. Thus, the structured model summary 1006 can be progressively updated using templates for different model components.

[0195] Figure 10E illustrates an exemplary fourth screen of the exemplary UI 1000, which allows the user to review the structured model summary 1006. In this example, the structured model summary 1006 has been updated to include templates for entities “vehicles”, “drivers”, “warehouses”, and “packages”; templates for decisions “find entity order” and “select location”; a template for the objective “minimize vehicle travel time”; and templates for business rules (also known as constraints) “multiple warehouses & time windows”, “driver work regulations”, and “vehicle capacity”.Each template displayed in the structured model summary 1006 can be selected by the user, and selecting a specific template may cause the UI 1000 to provide a pop-up window displaying a context-based explanation of that specific template (which was previously generated when retrieving the template). The UI 1000 can provide selectable options (not shown) to remove a specific template from the structured model summary and / or queries that can be added to the structured model summary. The UI 1000 also provides a selectable option 1026 for the user to approve the structured model summary. When the user approves the structured model summary, the interpretable template retrieval system 100 uses the structured model summary as a query to retrieve one or more base model templates.

[0196] Figure 10F shows an exemplary fifth screen of the exemplary UI 1000, which can be displayed after the interpretable template retrieval system 100 has retrieved one or more base model templates. In this example, the retrieved templates 1028 all display a title 1028a, a generated explanation 1028b, and a relevant domain indication 1028c. A user can (e.g., by clicking one of the displayed templates 1028) select one of the displayed base model templates 1028 and select option 1030 to approve the use of the selected base model template 1028 as the basis for building the optimized model.

[0197] Figure 10G shows an exemplary pop-up window 1032 that can provide more information about the base model templates 1028 shown in Figure 10F. For example, pop-up window 1032 can be displayed when one of the base model templates 1028 is selected but the user does not approve it. In this example, pop-up window 1032 makes a horizontal comparison between the structured model summary 1006 (e.g., the structured model summary 1006 approved by the user to query the base model template, e.g., as shown in Figure 10E) and the base model summary 1034 of one of the base model templates 1028 that has already been retrieved. Pop-up window 1032 provides an explanation of the base model template (e.g., a context-based explanation of the base model template, which may have been generated when the base model template was retrieved).

[0198] The pop-up window 1032 can also perform a visual comparison between the user-approved structured model summary 1006 and the base model summary 1034 of the base model template. For example, model components common to both the structured model summary 1006 and the base model summary 1034 can be highlighted in green; model components found in the structured model summary 1006 but not in the base model summary 1034 can be displayed in white; and model components found in the base model summary 1034 but not in the structured model summary 1006 can be displayed in red.Each model component displayed in pop-up window 1032 can be selected by the user to invoke another pop-up window (not shown) that provides a context-based explanation of the template corresponding to the selected model component. This allows the user to better understand the relevance of the retrieved base model template.

[0199] Returning to Figure 10F, after the user selects and approves one of the base model templates 1028, UI 1000 can progressively build an optimized model based on the base model template by querying and retrieving additional templates. This can be similar to the process of adding templates to a structured model summary, for example, as shown in the examples in Figures 10B to 10D. When the user is satisfied with the optimized model and no further additional templates are needed, the user can choose to approve the final optimized model and output the optimized model as a symbolic model. Contextual information can also be output along with annotations (automatically inserted by the interpretable template retrieval system 100 as described above) to provide documentation of the generated optimized model.

[0200] While exemplary UI 1000 has been illustrated and described, it should be understood that this is not limiting. For example, the interpretable template retrieval system 100 and / or incremental modeling method 600 may or may not have any UI. Alternatively, any UI that supports user natural language text input may be suitable for the examples of this invention.

[0201] In various examples, the present invention describes systems and methods capable of generating optimization models. Optimization models can be generated incrementally (or iteratively), with templates used as building blocks. Each template can be presented to the user through context-based interpretation, which allows the user to understand the meaning of the template in the context of a specific optimization problem.

[0202] Various examples of the present invention enable users to utilize optimization solvers by generating optimization models (which can be instantiated as model instances to be solved by the solver) without requiring the user to directly code the optimization model. Various examples of the present invention can accept natural language input as input, which may make it easier for users to access the solver.

[0203] Various examples of the present invention provide a flexible framework for generating optimization models. No fixed decision tree must be traversed in the process of reaching the final optimization model. Furthermore, various examples of this invention enable the optimization model to be modified and / or refined in a modular manner at different stages (e.g., by modifying, adding, or deleting templates). This allows for the generation of the optimization model in a more efficient manner. For example, users can participate in the entire model generation process, and instead of having to approve the entire generated optimization model (so that if changes are needed, the entire optimization model must be regenerated), users can review and approve the individual templates used to build the optimization model. During the model generation process, users can be guided by generating context-based explanations for each retrieved template.Therefore, the need to regenerate the optimization model due to user modifications can be reduced or avoided.

[0204] In some examples, contextual information providing the context of the optimization problem can be automatically annotated using information about the template used to generate the optimization model. The annotated contextual information can be used as documentation for the optimization model and a record of how the optimization model was generated. This self-documenting capability can help automatically generate supporting documentation for the optimization model, which includes both the context of the problem and a technical explanation of the optimization model. Automating the generation of such documentation helps reduce the amount of manual input required.

[0205] While examples of input tagged documents being unstructured multimodal documents have been described, it should be understood that the invention can also be applied to cases where the tagged documents are structured (e.g., following specific rules for defining the optimization problem) or contain only mathematical content, etc.

[0206] While the invention describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be appropriately omitted or changed. Where appropriate, one or more steps may be performed in an order other than that described.

[0207] While the invention has been described at least partially in terms of method, those skilled in the art will understand that the invention is also directed to various components for performing at least some aspects and features of the method, whether by hardware components, software, or any combination thereof. Accordingly, the technical solutions of the invention can be embodied in the form of a software product. Suitable software products can be stored in pre-recorded storage devices or other similar non-volatile or non-transient computer-readable media, including DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes instructions tangibly stored thereon that cause a processing device (e.g., a personal computer, server, or network device) to perform examples of the methods disclosed herein. Machine-executable instructions can be in the form of code sequences, configuration information, or other data that, when executed, cause a machine (e.g., a processor or other processing device) to perform the steps in the method according to the examples of the invention.

[0208] The invention may be practiced in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and are not restrictive. Features selected from one or more of the above embodiments can be combined to create alternative embodiments that are not explicitly described, and features suitable for such combinations will be understood within the scope of the invention.

[0209] All values ​​and subranges within the disclosed scope are also disclosed.Furthermore, while the systems, devices, and processes disclosed and shown herein may include a specific number of units / components, these systems, devices, and components may be modified to include more or fewer such units / components. For example, while any unit / component disclosed may be referred to as a single quantity, embodiments disclosed herein may be modified to include multiple such units / components. The subject matter described herein is intended to cover and encompass all appropriate technical changes. Instruction manual 27 / 27 pages 32 CN 121153030 A Figure 1 Figure 2 Instruction manual Figure 1 / 16 pages 33 CN 121153030 A Figure 3A Instruction manual Figure 2 / 16 pages 34 CN 121153030 A Figure 3B Figure 4A Instruction manual Figure 3 / 16 pages 35 CN 121153030 A Figure 4B Figure 5A Instruction manual Figure 4 / 16 pages 36 CN 121153030 A Figure 5B Instruction manual Figure 5 / 16 pages 37 CN 121153030 A Figure 6 Instruction manual Figure 6 / 16 pages 38 CN 121153030 A Figure 7 Instruction manual Figure 7 / 16 pages 39 CN 121153030 A Figure 8 Instruction manual Figure 8 / 16 pages 40 CN 121153030 A Figure 9 Instruction manual Figure 9 / 16 pages 41 CN 121153030 A Figure 10A Instruction Manual Appendix 10 / 16 Page 42 CN 121153030 A Figure 10B Instruction Manual Appendix 11 / 16 Page 43 CN 121153030 A Figure 10C Instruction Manual Appendix 12 / 16 Page 44 CN 121153030 A Figure 10D Instruction Manual Appendix 13 / 16 Page 45 CN 121153030 A Figure 10E Instruction Manual Appendix 14 / 16 Page 46 CN 121153030 A Figure 10F Instruction Manual Appendix 15 / 16 Page 47 CN 121153030 A Figure 10G Instruction Manual Appendix 16 / 16 Page 48 CN 121153030 A.

Claims

1. A computing system, characterized in that, The computing system includes: Processing unit, configured to execute computer-readable instructions to cause the system to perform the following operations: Receive context information from the user equipment, wherein the context information includes a textual description of the optimization problem in natural language; The structured model summary data structure is updated by retrieving one or more templates of one or more model components from a template database, wherein each module in the one or more templates is retrieved using a fine-grained query generated from the context information; and updating the structured model summary data structure using metadata associated with each of the retrieved one or more templates. The structured model summary data structure is used as a query to the template database to retrieve the basic model template; The symbolic model associated with the base model template is updated by retrieving one or more additional templates for one or more additional model components from the template database, wherein each of the one or more templates is retrieved using an additional fine-grained query; and updating the symbolic model using the symbolic content associated with each of the one or more retrieved templates. The symbolic model is output as the generated optimized model.

2. The computing system according to claim 1, characterized in that, The updated structured model summary data structure includes updating the context information using metadata associated with the retrieved templates (one or more), wherein the updated context information is output.

3. The computing system according to claim 1 or 2, characterized in that, The updating of the structured model summary data structure is performed iteratively, and the structured model summary data structure is updated in each iteration using metadata associated with a retrieved template of a model component.

4. The computing system according to claim 3, characterized in that, The update of the symbol model is performed iteratively, and the symbol model is updated in each iteration using symbolic content associated with a retrieved template of a model component.

5. The computing system according to any one of claims 1 to 4, characterized in that, The updated structured model summary data structure includes generating a context-based explanation for each of the retrieved templates, wherein the context-based explanation is presented via the user device.

6. The computing system according to any one of claims 1 to 5, characterized in that, The updated structured model summary data includes: presenting the at least one fine-grained query through the user equipment, and receiving the approved at least one fine-grained query through the user equipment before retrieving at least one template using the at least one fine-grained query.

7. The computing system according to any one of claims 1 to 6, characterized in that, The processing unit is also configured to execute computer-readable instructions to cause the system to perform the following operations: After retrieving the base model template, an explanation of the generation of the base model template is presented through the user device, wherein the generated explanation is generated from the context information and metadata associated with the base model template.

8. The computing system according to any one of claims 1 to 7, characterized in that, The update of the symbol model includes: presenting the at least one additional fine-grained query through the user equipment, and receiving the approved at least one additional fine-grained query through the user equipment before retrieving at least one template using the at least one additional fine-grained query.

9. The computing system according to any one of claims 1 to 8, characterized in that, The at least one fine-grained query is generated based on the initial query received from the user equipment.

10. The computing system according to any one of claims 1 to 9, characterized in that, The processing unit is also configured to execute computer-readable instructions to cause the system to perform the following operations: A user interface (UI) is provided to the user equipment, wherein the context information is received from the user equipment as natural language input through the UI.

11. The computing system according to any one of claims 1 to 10, characterized in that, The processing unit is further configured to execute computer-readable instructions to enable the system to implement a fine-grained query generator, wherein the fine-grained query generator includes: The first natural language processing (NLP) encoder is trained to encode context fragments segmented from the context information into context vectors and to encode the initial query into a query vector, wherein the context vector and the query vector are encoded into a common vector space; A first similarity module is used to select the at least one selected context segment based on the similarity between the query vector and a context vector encoded from at least one selected context segment. The first NLP generative model is trained to generate at least one refined query based on the initial query and the at least one selected context fragment.

12. The computing system according to claim 11, characterized in that, The processing unit is further configured to execute computer-readable instructions to enable the system to implement a template retrieval device, wherein the template retrieval device includes: The second NLP encoder is trained to encode the at least one fine-grained query into a fine-grained query vector and to encode the metadata associated with each of the one or more templates into one or more template vectors, wherein the one or more template vectors and the fine-grained query vectors are encoded into a common vector space. The second similarity module is used to select the at least one selected template based on the similarity between the fine query vector and the template vector encoded from the metadata associated with the at least one selected template to be retrieved.

13. The computing system according to claim 12, characterized in that, The processing unit is further configured to execute computer-readable instructions to enable the system to implement an interpreter generator, wherein the interpreter generator includes: The second NLP generative model is trained to generate a context-based interpretation for the at least one selected template based on the metadata associated with the at least one selected template and the fine-grained query.

14. A method, characterized in that, The method includes: Receive context information from the user equipment, wherein the context information includes a textual description of the optimization problem in natural language; The structured model summary data structure is updated by retrieving one or more templates of one or more model components from a template database, wherein each of the one or more templates is retrieved using a corresponding fine-grained query generated from the context information; and updating the structured model summary data structure using metadata associated with each of the one or more retrieved templates. The structured model summary data structure is used as a query to the template database to retrieve the basic model template; The symbolic model associated with the base model template is updated by retrieving one or more additional templates for one or more additional model components from the template database, wherein each of the one or more templates is retrieved using an additional fine-grained query; and updating the symbolic model using the symbolic content associated with each of the one or more retrieved templates. The symbolic model is output as the generated optimized model.

15. The method according to claim 14, characterized in that, The updated structured model summary data structure includes updating the context information using metadata associated with the retrieved templates (one or more), wherein the updated context information is output.

16. The method according to claim 14 or 15, characterized in that, The updating of the structured model summary data structure is performed iteratively, and the structured model summary data structure is updated in each iteration using metadata associated with a retrieved template of a model component.

17. The method according to claim 16, characterized in that, The update of the symbol model is performed iteratively, and the symbol model is updated in each iteration using symbolic content associated with a retrieved template of a model component.

18. The method according to any one of claims 14 to 17, characterized in that, The updated structured model summary data structure includes generating a context-based explanation for each of the retrieved templates, wherein the context-based explanation is presented via the user device.

19. The method according to any one of claims 14 to 18, characterized in that, The updated structured model summary data includes: presenting the at least one fine-grained query through the user equipment, and receiving the approved at least one fine-grained query through the user equipment before retrieving at least one template using the at least one fine-grained query.

20. The method according to any one of claims 14 to 19, characterized in that, The method further includes: After retrieving the base model template, an explanation of the generation of the base model template is presented through the user device, wherein the generated explanation is generated from the context information and metadata associated with the base model template.

21. The method according to any one of claims 14 to 20, characterized in that, The update of the symbol model includes: presenting the at least one additional fine-grained query through the user equipment, and receiving the approved at least one additional fine-grained query through the user equipment before retrieving at least one template using the at least one additional fine-grained query.

22. The method according to any one of claims 14 to 21, characterized in that, The at least one fine-grained query is generated based on the initial query received from the user equipment.

23. The method according to any one of claims 14 to 22, characterized in that, The method further includes: A user interface (UI) is provided to the user equipment, wherein the context information is received from the user equipment as natural language input through the UI.

24. The method according to any one of claims 14 to 23, characterized in that, The method further includes executing a fine-grained query generator, wherein the fine-grained query generator includes: The first natural language processing (NLP) encoder is trained to encode context fragments segmented from the context information into context vectors and to encode the initial query into a query vector, wherein the context vector and the query vector are encoded into a common vector space; A first similarity module is used to select the at least one selected context segment based on the similarity between the query vector and a context vector encoded from at least one selected context segment. The first NLP generative model is trained to generate at least one refined query based on the initial query and the at least one selected context fragment.

25. The method according to claim 24, characterized in that, The method further includes executing a template retriever, wherein the template retriever includes: The second NLP encoder is trained to encode the at least one fine-grained query into a fine-grained query vector and to encode the metadata associated with each of the one or more templates into one or more template vectors, wherein the one or more template vectors and the fine-grained query vectors are encoded into a common vector space. The second similarity module is used to select the at least one selected template based on the similarity between the fine query vector and the template vector encoded from the metadata associated with the at least one selected template to be retrieved.

26. The method according to claim 25, characterized in that, The method further includes executing an interpretation generator, wherein the interpretation generator comprises: The second NLP generative model is trained to generate a context-based interpretation for the at least one selected template based on the metadata associated with the at least one selected template and the fine-grained query.

27. A non-transitory computer-readable medium for storing instructions, characterized in that, The instructions can be executed by a processing unit in a computing system to cause the computing system to perform the method according to any one of claims 14 to 26.

Images