Method and system for model generation and instantiation of optimization models based on tagged documents

Abstract

Methods and systems for generating a symbolic model from a tagged document and instantiating a model instance from the symbolic model are described. A tagged document containing human language content and mathematical content is parsed into a symbolic model containing only symbolic codes representing an optimization problem. The tagged document is parsed to extract a tagged statement, and then the tagged statement is processed into a mathematical content span, any metadata entity, and any relationship between any metadata entity and the mathematical content span. And processing the mathematical content span into a mathematical content analysis tree. The mathematical content parse tree is converted to symbol code of the symbol model using any relationship between the metadata entity and the mathematical content span. The symbolic model may be instantiated using a data definition.

Description

Technical Field

[0001] The present invention relates to a method and system for parsing a markup document to automatically generate computer code representing a model instance, wherein the model instance can be input to an optimization solver to solve an optimization problem. Background Art

[0002] Many real-world problems are optimization problems, such as how to minimize the use of limited resources, or how to maximize the throughput of a system. In many cases, these optimization problems are complex (e.g., have many different constraints, many different conditions, many different variables, etc.) and cannot be practically solved by humans. Optimization solvers (also simply called "solvers") are software tools developed to solve such optimization problems.

[0003] The traditional process for solving optimization problems involves human experts creating a tagged document that formulates one or more optimization objectives, constraints, etc. The tagged document involves the use of both natural language (i.e., text content understandable to humans) and mathematical formulas. Then, another human expert (e.g., an operations research (OR) expert) converts the tagged document into an optimization model and instantiates the model into a model instance using actual numerical data. The model instance is in the form of a computer-readable code that can be input to a solver. If the optimization problem is adjusted (e.g., by adding or deleting constraints), the OR expert must make corresponding changes to the code of the model and the model instance. Similarly, if a new model instance is needed (e.g., when new data is available), the OR expert must generate code for the new model instance. This process is inefficient in terms of time and resources.

[0004] Requiring an OR expert also limits the practical use of the solver, since the OR expert may become a bottleneck.

[0005] It would be useful to provide a solution that could improve the accessibility of solvers and / or improve the efficiency of solving optimization problems. Summary of the invention

[0006] In various examples, the present invention describes methods and systems for generating a symbolic model from a tagged document describing an optimization problem, the tagged document may be an unstructured multimodal tagged document. The present invention also describes methods and systems for instantiating the symbolic model into a model instance that may be input to a solver to solve the optimization problem. The disclosed methods and systems are capable of converting a tagged document into a model instance that may be input to a solver without requiring the expertise of a human OR expert.

[0007] In some examples, the present invention describes the generation of symbolic models, which can be stored in a symbolic model database. This provides a technical advantage that model instantiation can be performed efficiently by retrieving the already generated symbolic models from the database and instantiating the models using data values, without having to generate the models each time. In addition, examples of the present invention enable the semantic meaning of each symbolic model to be stored in the symbolic model database, which provides the advantage that the symbolic model database can be directly searched to retrieve the desired symbolic model.

[0008] Examples of the present invention can generate model instances for various solvers (which may have different configurations) without requiring human experts to manually code each model instance according to the requirements of each solver. This provides better flexibility and scalability, and improves efficiency.

[0009] Examples of the present invention can decouple the generation of symbolic models from the instantiation of model instances. This can provide the following technical advantages: different model instances can be instantiated to adapt to different data definitions and / or different solvers without having to generate a new symbolic model each time. This can simplify the process of updating data, adapting to new solvers, debugging code, or exploring different data scenarios.

[0010] In one exemplary aspect, the present invention describes a method, comprising: receiving a markup document representing an optimization problem, the markup document comprising human language content and mathematical content; parsing the markup document into a symbolic model containing only symbolic codes representing the optimization problem by: extracting at least one markup statement from the markup document; for the at least one markup statement, extracting at least one corresponding mathematical content span, and extracting any corresponding metadata entities and any relationship between the at least one corresponding mathematical content span and any corresponding metadata entity; for the at least one mathematical content span, generating a corresponding mathematical content parse tree; using any relationship between the at least one corresponding mathematical content span and any corresponding metadata entity, converting the corresponding mathematical content parse tree into symbolic codes, the symbolic codes forming the symbolic model; the method also includes: outputting the symbolic model associated with semantic metadata containing the at least one corresponding metadata entity.

[0011] In one example of the aforementioned exemplary aspect of the method, the method may further include: implementing a declaration extractor using a first trained neural network to extract the at least one markup declaration from the markup document; implementing a named entity recognition and relationship extractor using a second trained neural network to extract the at least one corresponding mathematical content span, any corresponding metadata entities, and any relationship between the at least one corresponding mathematical content span and any corresponding metadata entities.

[0012] In one example of the aforementioned exemplary aspect of the method, the first trained neural network and the second trained neural network may be based on a pre-trained natural language processing neural network, respectively.

[0013] In an example of the aforementioned exemplary aspect of the method, the method may also include: providing a user interface for receiving the markup document and for displaying at least one symbolic parameter of the symbolic model, the at least one symbolic parameter corresponding to the at least one mathematical content span, and the at least one symbolic parameter being displayed together with any related corresponding metadata entities.

[0014] In one example of the aforementioned exemplary aspect of the method, the method may further include: receiving input to edit the at least one symbol parameter.

[0015] In one example of the aforementioned exemplary aspect of the method, the symbol model associated with the semantic metadata may be stored in a symbol model database.

[0016] In an example of the aforementioned exemplary aspect of the method, the method may also include: obtaining at least one data definition; generating at least one data definition mapping, the at least one data definition mapping mapping the at least one data definition to at least one symbolic parameter of the symbolic model; processing the symbolic model using the at least one data definition mapping and the at least one data definition to generate a model instance; and outputting the model instance to be solved by a solver.

[0017] In one example of the aforementioned exemplary aspect of the method, the method may further include: obtaining a solver configuration; wherein the model instance may be generated according to the solver configuration.

[0018] In one example of some of the aforementioned exemplary aspects of the method, the method may further include: identifying a difference between the at least one data definition and the at least one symbol parameter; and outputting a notification indicating the difference.

[0019] In one example of some of the aforementioned exemplary aspects of the method, the symbol model may be stored in a database, and the method may further include retrieving the symbol model from the database in response to a query.

[0020] In some exemplary aspects, the present disclosure describes a computing system comprising a processing unit for executing computer-readable instructions to cause the system to perform a method of any of the foregoing exemplary aspects of the method.

[0021] In another exemplary aspect, the present invention describes a non-transitory computer-readable medium having machine-executable instructions stored thereon, wherein the instructions, when executed by a processing unit of a device, cause the device to perform a method of any of the foregoing exemplary aspects of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Reference is now made by way of example to the accompanying drawings which illustrate exemplary embodiments of the present application, in which:

[0023] Figure 1 is a block diagram of a simplified exemplary system provided by examples of the present invention, which can be used to generate a symbolic model and instantiate a model instance;

[0024] Figure 2 is a block diagram of an exemplary computing system that may be used to implement examples of the present invention;

[0025] Figure 3A is a block diagram showing components of an exemplary symbol model generator provided by examples of the present invention, which can be used to Figure 1 system;

[0026] FIG. 3B to FIG. 3D The exemplary annotated training data provided by the example of the present invention is shown, and the exemplary annotated training data can be used for training Figure 3A Neural network for symbolic model generator;

[0027] Figure 3E is a block diagram showing components of an exemplary named entity recognition and relationship extractor provided by an example of the present invention, which can be used to Figure 3A Symbolic model generator for ;

[0028] Figure 3F An exemplary mathematical content parse tree provided by an example of the present invention is shown. The exemplary mathematical content parse tree can be represented by Figure 3A Subsystem generation of the symbolic model generator;

[0029] Figure 4 An example of the present invention provides Figure 3A An example of how the symbolic model generator processes a marked-up document into a symbolic model;

[0030] Figure 5 The example of the present invention provides a method that can be performed by Figure 1 An exemplary user interface provided by the system;

[0031] Figure 6 is a block diagram showing components of an exemplary model instantiator provided by examples of the present invention, which can be used to Figure 1 system;

[0032] Figure 7 is a flow chart illustrating an exemplary method for generating a symbol model from a markup document provided by an example of the present invention;

[0033] Figure 8 is a flow chart illustrating an exemplary method for instantiating a model instance from a symbolic model provided by examples of the present invention.

[0034] Similar reference numerals may be used in different drawings to represent similar components. DETAILED DESCRIPTION

[0035] To aid in understanding the present invention, some background discussion of optimization problems and solvers is first provided.

[0036] Optimization problems are often described mathematically by domain experts in the format of a markup document. Domain experts can use mathematical symbols to represent decision variables (e.g., unknown variables to be solved) and data parameters (e.g., parameters whose values ​​can be filled with real-world data), and can use mathematical formulas to define the optimization objective (e.g., the function to be maximized or minimized) and applicable constraints (e.g., inequality and / or equality constraints). Human-understandable text (such as English text) is used to define the meaning of symbols and formulas in the context of the optimization problem. A markup document can also be called a multimodal document because it involves both human-understandable text and computer-readable symbols (or "markups"). Some commonly used markup languages ​​for creating markup documents include Markdown, TeX, LaTeX, MathML, and OpenMath, among others. Each markup language defines certain syntax and symbols that domain experts can use to intersperse mathematical content with natural language content.

[0037] In order for an optimization problem to be understood and solved by an optimization solver (hereinafter referred to as a "solver" or "software solver"), the markup document must be converted into a computer-readable code. This involves the skills of an operations research (OR) expert to convert the markup document into a symbolic model and then instantiate the model. A symbolic model is a symbolic and context-free representation of an optimization problem encoded in a computational programming language or modeling language. An optimization problem may have one or more objectives, and one or more constraints. In many optimization problems, there is one objective and multiple constraints, while in other optimization problems, there may be multiple objectives and / or a single constraint. The present invention may refer to "objective" in the singular and "constraints" in the plural, but the use of the singular or plural is not intended to be limiting. A symbolic model may contain the mathematical objectives and constraints of the optimization problem, expressed as mathematical functions of symbols that represent data parameters and decision variables. The symbolic model may omit the semantic meaning of mathematical symbols and formulas, so that the actual real-world relevance of the symbols and formulas may not be easily apparent from the symbolic model alone.

[0038] It should be noted that the symbolic model does not contain actual data values. The process of importing data into the symbolic model can be called model instantiation. Model instantiation is a task usually performed by OR experts. When the data parameters of the symbolic model are assigned actual data values ​​(for example, from a real-world scenario), the symbolic model is instantiated and a model instance is created, which is a specific instance of the symbolic model in the form of computer-readable code that can be input to a solver to generate a solution to the optimization problem. There are various solvers that can be used to solve different optimization problems, and each solver may require the model instance to provide different specific structured inputs (for example, specific keywords, specific code sequences, etc.).

[0039] Therefore, the skills of an OR expert are important in translating the optimization from a human-understandable language (e.g., as represented by a markup document) into a computer-readable code (e.g., a model instance). Furthermore, because the symbolic model instances are stripped of semantic meaning, the symbolic models and model instances may be difficult to understand by people who are not OR experts. Therefore, it may be difficult for domain experts to verify whether the optimization problem has been accurately represented in the symbolic model or machine instance. Similarly, it may be difficult for domain experts to make even minor changes to the optimization problem without the help of an OR expert.

[0040] Some attempts have been made to simplify this process, but no solution to date has been able to successfully convert labeled documents into model instances without requiring a lot of human input. For example, some existing methods can only generate model instances when provided with input that is structured in a very specific and limited way (for example, it must contain only mathematical content, must use specific keywords, or the optimization problem must be defined in a specific order). As a result, the process is still inefficient and still requires some OR expertise to manually process or structure the labeled documents. In addition, some existing methods generate model instances directly from labeled documents and data definitions without the intermediate step of generating a symbolic model. This means that if the data definition changes (for example, there is more up-to-date real-world data to be input into the model instance), the labeled documents need to be reprocessed from scratch.

[0041] In various examples, the present invention describes methods and systems that can convert unstructured technical documents (e.g., markup documents) containing both natural language text and mathematical content into model instances that can be input to a solver. In particular, the present invention describes methods and systems that generate symbolic models from markup documents as an intermediate step. For example, compared to existing methods that generate model instances directly from markup documents, symbolic models can be stored for future use and can be more easily processed into new model instances to accommodate new data definitions or changes to model constraints or objectives.

[0042] Figure 1 is a schematic diagram showing an exemplary optimization problem modeling system 100. The optimization problem modeling system 100 can be implemented in a single physical machine or device (e.g., implemented as a single computing device, such as a single workstation, a single server, etc.), or can be implemented using multiple physical machines or devices (e.g., implemented as a server cluster). For example, the optimization problem modeling system 100 can be implemented as a virtual machine or a cloud-based service (e.g., implemented using a cloud computing platform that provides a virtualized pool of computing resources). In some examples, the optimization problem modeling system 100 can provide a cloud-based service accessible to a user device.

[0043] In this example, the optimization problem modeling system 100 communicates with the solver 10 (e.g., via a wired or wireless network) and, optionally, with the symbolic model database 250. The symbolic model database 250 may be external to the optimization problem modeling system 100 (as shown), or may be part of the optimization problem modeling system 100. Optionally, the optimization problem modeling system 100 may also communicate with one or more user devices (not shown). In some examples, the optimization problem modeling system 100 may not communicate directly with the solver 10. For example, instead of the optimization problem modeling system 100 providing the model instance directly to the solver 10, the optimization problem modeling system 100 may output the model instance (e.g., as a computer readable code stored in a tangible medium, such as a removable memory), and may provide the model instance to the solver 10 at some later time.

[0044] In the example shown, the optimization problem modeling system 100 includes a symbolic model generator 200 and a model instantiator 300 as subsystems. In other examples, the symbolic model generator 200 and the model instantiator 300 may be independent systems that communicate with each other. In still other examples, the optimization problem modeling system 100 may provide the functions of symbolic model generation and model instantiation by itself without the subsystems 200 and 300. It should be understood that the optimization problem modeling system 100 may be implemented using a variety of hardware and software and is not intended to be limited to Figure 1 Example shown.

[0045] The symbolic model generator 200 receives as input a markup document (which may be in digital format) representing an optimization problem. The optimization problem may describe a real-world decision scenario and may be represented by mathematical formulas and symbols (which may model the real-world scenario). There may be different types of optimization problems that may be solved by examples of the present invention, such as linear programming, quadratic programming, mixed integer programming, etc.

[0046] For example, a tagged document can be provided from a user device. As described above, a tagged document can be an unstructured multimodal document, which means that the tagged document includes both human language content and mathematical content, and the content does not necessarily follow a defined structure. The tagged document uses a markup language (e.g., using TeX, LaTeX, Markdown, or XML) to describe the optimization problem using some combination of natural human language (e.g., human-understandable text) and mathematical representation (e.g., mathematical formulas and symbols). The tagged document is processed by the symbolic model generator 200 to generate a symbolic model. The symbolic model describes the optimization problem using only mathematical representations (i.e., without textual descriptors). As will be further described below, the symbolic model generator 200 can extract semantic metadata and associate the semantic metadata with the generated symbolic model. Optionally, the symbolic model and the associated semantic metadata can be stored in the symbolic model database 250.

[0047] The symbolic model is received as input to the model instantiator 300. The symbolic model may be provided directly to the model instantiator 300 by the symbolic model generator 200, or may alternatively be retrieved from the symbolic model database 250. The model instantiator 300 also receives as input one or more data definitions (e.g., from a user device) that provide sets of defined values ​​and numerical values ​​of parameters for providing symbolic constraints and objectives of the symbolic model. Optionally, the model instantiator 300 may also receive (e.g., from a user device) information about a solver configuration. For example, the model instantiator 300 may receive an indication or selection of a solver application programming interface (API) to be used. A solver API is a software tool that provides an interface with a particular solver 10. The solver API provides a tool that can be used by the model instantiator 300 to generate a model instance. In some examples, if the model instantiator 300 does not receive information about a solver configuration, a default solver API may be used.

[0048] The model instantiator 300 generates a model instance. The model instance can be in the form of a computer-readable code (e.g., using a coding language that can be executed by the solver 10) and can be processed by the solver 10. The model instance is a computer-readable representation of an optimization problem, in which the values ​​of parameters and constraints (defined in symbolic form in the symbolic model) have been defined in digital form using data definitions. It should be understood that the model instantiator 300 can generate different model instances for the same symbolic model using different data definitions. The model instance is provided as an input to the solver 10, which processes the model instance to generate a solver result that solves the objective function of the optimization problem while complying with the constraints. The solver result can be used as a solution to the optimization problem.

[0049] As described above, the optimization problem modeling system 100 may be implemented using different hardware configurations. An exemplary computing system that may be used to implement the optimization problem modeling system 100 is now described.

[0050] Figure 2 4 is a block diagram of a simplified exemplary computing system 400 that may be used to implement the optimization problem modeling system 100 in some embodiments. For example, the computing system 400 may represent a server or workstation. As described above, the optimization problem modeling system 100 may be implemented in other hardware configurations, including implementations using multiple computer systems. Figure 2 A single instance of each component is shown, but multiple instances of each component may exist in computing system 400. Computing system 400 may be used to execute instructions for training a neural network and / or for executing a trained neural network, as discussed further below.

[0051] In this example, the computing system 400 includes at least one processing unit 402, which can be a processor, a microprocessor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processing unit, a graphics processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), a hardware accelerator, a combination thereof, or other such hardware structures.

[0052] Computing system 400 may include an input / output (I / O) interface 404 that may support connection with input devices and / or output devices (not shown).

[0053] The computing system 400 may include a network interface 406 for wired or wireless communication with other computing systems (e.g., symbolic model database 250, solver 10, user equipment, etc.). The network interface 406 may include a wired link (e.g., Ethernet cable) and / or a wireless link (e.g., one or more antennas) for intra-network and / or inter-network communication. The network interface 406 may also enable the computing system 400 to send a generated report to another computing system (e.g., to a user equipment).

[0054] The computing system 400 may include a storage unit 408 , which may include a mass storage unit such as a solid-state drive, a hard disk drive, a magnetic disk drive, and / or an optical disk drive.

[0055] The computing system 400 may include a memory 410, which may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The non-transitory memory 410 may store instructions executed by the processing unit 402, for example, to perform the exemplary embodiments described herein. For example, the memory 410 may store instructions 412 for implementing the optimization problem modeling system 100 and any of the methods disclosed herein. The memory 410 may also store neural network parameters, which may be parameters learned by training a neural network, such as in an example where the optimization problem modeling system 100 is implemented using a neural network. The memory 410 may include other software instructions, such as software instructions for implementing an operating system and other applications / functions.

[0056] Additionally or alternatively, the computing system 400 may execute instructions from an external memory (e.g., an external drive in wired or wireless communication with a server), or executable instructions may be provided by a transient or non-transient computer-readable medium. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory.

[0057] Reference again Figure 1 The optimization problem modeling system 100 helps improve the accessibility and usability of the solver by automatically parsing a tagged document (which is an unstructured multimodal document with a mixture of text and mathematical content) into a symbolic model (which contains only mathematical content) using a symbolic model generator 200, and then processing the symbolic model using a model instantiator 300 to generate a model instance (which is a computer-readable code) that can be input to the solver 10. The tagged document can be processed into a model instance without requiring the knowledge of an OR expert.

[0058] It should be noted that the generation of the symbolic model by the symbolic model generator and the instantiation of the model instance by the model instantiator 300 can be decoupled, which means that the data definition can be defined asynchronously with the creation of the markup document and the generation of the symbolic model at any time. In addition, the markup document and the data definition can be updated independently at any time, or a new solver 10 can be selected at any time and a new model instance can be generated. This can make the process of solving the optimization problem more efficient.

[0059] The details of the symbolic model generator 200 are discussed first.

[0060] The symbolic model generator 200 performs the function of converting a markup document into a symbolic model. The symbolic model generator 200 can receive a multimodal unstructured markup document as input without requiring the markup document to follow a specific structure or use a specific language. Non-OR experts can generate markup documents using any suitable markup language (e.g., TeX, LaTeX, Markdown, XML, etc.) using a combination of natural human language text (e.g., English language text formatted as paragraphs, bullet points, section headings, etc.) and mathematical content (e.g., symbols, equations, etc.).

[0061] The symbolic model generator 200 can use a hierarchical parsing technique to process unstructured tagged documents without the intervention of OR experts or other personnel, and without the use of specific structured inputs or keywords. The hierarchical parsing technique can parse the tagged documents by first detecting the high-level components of the symbolic model (e.g., natural language text or mathematical content describing constraints), and then further parsing the low-level building blocks (e.g., symbolic variables and / or parameters). The high-level parsing can be used to classify the unstructured input into one of several defined optimization problem component categories (e.g., constraints, objectives, variables, etc.). Then, a specialized low-level parsing can be performed for each defined optimization problem component category.

[0062] The symbolic model generator 200 supports the extraction of semantic metadata from markup documents and associating with symbolic models. The metadata in the markup documents may include human language text that describes the symbolic components of the optimization problem and gives them meaning. Traditionally, such metadata is used to help OR experts understand the context of the optimization problem, but is not included in the model designed by the OR experts. This makes it difficult for non-OR experts to understand the model or adjust the model. The disclosed symbolic model generator 200 performs operations to automatically parse and extract semantic metadata from the markup documents, and stores the extracted semantic metadata associated with the corresponding symbolic components. This helps to improve the user accessibility of the symbolic model. Similarly, the symbolic models stored in the symbolic model database 250 can be searched using the associated metadata.

[0063] Figure 3Ais a block diagram illustrating an exemplary implementation of the symbol model generator 200. Although the symbol model generator 200 is illustrated with certain functional blocks, it should be understood that this is not intended to be limiting.

[0064] The symbolic model generator 200 receives a markup document as input. Hierarchical parsing of the markup document is performed by a declaration extractor 202 (which can be considered a high-level parser), followed by a named entity recognition (NER) and relation extractor 204 (which can be considered a mid-level parser), followed by a mathematical content parser 206 (which can be considered a low-level parser). Finally, a tree-to-definition converter 208 converts the parsed content into a symbolic model.

[0065] The declaration extractor 202 performs content segmentation on the markup document to extract content fragments corresponding to the markup declaration. A markup declaration is a fragment of symbols and optional text that can be classified into one of several defined declaration types, namely parameters, sets, decision variables, constraints, or objectives. In the present invention, parameters refer to constants (usually expressed as symbolic parameters) used to define constraints and objectives; sets refer to all possible values ​​of the index of a decision variable or parameter; decision variables are variables that are changed to achieve the goal of an optimization problem (usually, solving an optimization problem involves finding the optimal value of a decision variable); constraints are mathematical expressions that define the boundaries or restrictions of an optimization problem (for example, the number of people must be a non-negative integer, the maximum weight of cargo that can be carried on a truck, etc.); objectives are mathematical expressions that define the goal of an optimization problem (for example, minimizing costs, maximizing the amount of nutrients, etc.).

[0066] Claim extractor 202 may extract these claims using any suitable content segmentation technique. For example, claim extractor 202 may be a rule-based parser that identifies and segments claims from a markup document using defined rules of grammar, mathematics, etc. In another example, claim extractor 202 may be implemented using a text segmentation neural network that has been trained to recognize defined claim types and extract the claims.

[0067] For example, the claim extractor 202 can be a neural network based on pre-trained bidirectional encoder representations from transformers (BERT). The claim extractor 202 can be trained using a training data set including unstructured tagged documents, where the content has been segmented and labeled as one of the defined claim types. For example, hierarchical sequence labeling techniques can be used to annotate unstructured tagged documents to generate a training data set. The training data set can include labels that identify semantic metadata, metadata tags, and mathematical content entities. Labels can also be nested within labels, for example, mathematical content entities can be labeled, and left, right, and constraint direction labels can also be found (or "nested") within the mathematical content entity labels.

[0068] Figure 3B An example of annotated training data that may be used to train the claim extractor 202 is shown.

[0069] like Figure 3B As shown, a portion of the markup document 502 has been annotated to include a markup statement 504 (indicated by a dashed box). For example, the markup statement 504 may be annotated by identifying the first and last characters of the markup statement 504. Other non-annotation content ( Figure 3B The dashed box in the figure is not indicated) is not a markup declaration.

[0070] A pre-trained BERT-based neural network can be trained on this annotated training dataset using any suitable training method (e.g., using the Adam optimizer). After sufficient training, the learned neural network weights can be stored in local memory, and the trained neural network can be used as the claim extractor 202. In some examples, the neural network can be fine-tuned and validated. For example, fine-tuning and validation can be performed by extracting claims from a validation dataset using a neural network and then minimizing a cross-entropy loss that includes the error in entity prediction and the error in predicting metadata or mathematical content entities. The two losses can be combined using adjustable hyperparameters that adjust the weights of each loss. The neural network weights with the best performance on the validation dataset can be further evaluated using the entity-level micro-averaged F1 score on the retained test dataset. Other methods of learning and fine-tuning natural language processing (NLP) neural networks that can be used to implement the claim extractor 202 may be applicable in the context of the present invention.

[0071] Implementation of claim extractor 202 using a trained neural network to segment content may enable efficient parsing of unstructured markup documents without requiring the markup documents to follow a strict structure.

[0072] The output from the declaration extractor 202 is one or more marked declarations that have been segmented from the markup document. Each marked declaration belongs to one of the defined declaration types (parameter, set, decision variable, constraint, or goal), includes mathematical content (e.g., mathematical operators and / or symbols), and optionally, may include natural language text (e.g., English language text). For example, a marked declaration may be a statement that defines the meaning of a variable (e.g., "The first discrete variable is $P_{t}$ which is the production (in units) in month$t$ where $t\inT$."). It should be noted that not all content in a markup document may be a marked declaration. For example, a markup document may include text content, such as a title, introduction, etc., which are not declarations.

[0073] Each of the one or more markup declarations is then processed by a NER and relationship extractor 204. The NER and relationship extractor 204 performs the joint task of detecting metadata entities (if any) and mathematical content entities, and (if metadata entities are detected) extracting relationships between the detected metadata entities and the mathematical content entities. In general, NER may refer to the task of annotating content segments using annotations from a predefined list of available entity annotations, while relationship extraction may refer to the task of generating relationships between annotated content segments from a predefined list of possible relationships.

[0074] A metadata entity refers to a segment of human language content (e.g., English language text) identified as metadata; a mathematical content entity refers to a segment of content (e.g., symbolic content) identified as mathematical content. A mathematical content entity includes a string of one or more mathematical symbols or characters. A mathematical content entity can be characterized by its span, which describes the location of the entity within the statement and the length of the entity. Therefore, if human language content is present in the statement, the NER and relationship extractor 204 segments each tagged statement into mathematical content (e.g., symbols, equations, etc.), metadata (e.g., text descriptors), and (if metadata is present in the statement) also generates relationship data that associates the metadata entity with the corresponding mathematical content.

[0075] Metadata entities can be classified into metadata tags and semantic metadata. In general, each metadata entity provides some information about the meaning of the relevant mathematical content entity in the declaration. Metadata tags are texts that describe the information needed to build a symbolic model, such as text that defines variable types (e.g., discrete variables, integers, continuous, etc.), or text that defines the meaning of a goal (e.g., minimize or maximize). Semantic metadata is text that provides the meaning and definition of mathematical content, such as set definitions, variable definitions, goal names, constraint names, constraint properties (e.g., constraint direction, right-hand expression, etc.), and model properties (e.g., the number of variables, the number of constraints, etc.). In general, semantic metadata can provide contextual information to help human users understand the components of a symbolic model. Therefore, metadata tags or semantic metadata can provide humans with information to understand the relevant mathematical content entities, and can also provide information about how the relevant mathematical content entities should be defined in the symbolic model. It should be understood that a declaration can contain only mathematical content without any metadata tags or semantic metadata. For example, a declaration can contain only mathematical content whose symbolic variables have been defined by metadata in a previous declaration in a markup document.

[0076] The relationship data may be in the form of a link (e.g., a vector) that associates the metadata entity with the related mathematical content entity in the statement. It should be noted that in unstructured markup documents, it may be difficult to identify the relationship between the metadata entity and the mathematical content entity using a grammar-based or rule-based approach. Implementing the NER and relationship extractor 204 using a trained neural network that may be based on NLP can make the association between the metadata entity and the mathematical content entity more accurate, while allowing human domain experts greater flexibility in writing markup documents.

[0077] For example, a markup statement such as “The first discrete variable is $P_{t}$ which is the production (in units) in month $t$ where $t\inT$” may be parsed by the NER and relation extractor 204 into the metadata tag “discrete variable”, the semantic metadata “the production (in units) in month $t$”, and the mathematical content “$P_{t}$” and “$t\inT$”. Segments of mathematical content may be identified as mathematical content entities and characterized using mathematical content spans, which define the beginning and end of a single continuous segment of mathematical content. Mathematical content entities may be short (e.g., a single variable) or long (e.g., a constraint equation). The NER and relation extractor 204 may also associate the metadata tag “discrete variable” and the semantic metadata “the production (in units) in month $t$” with the mathematical content “$P_{t}$”. Furthermore, the mathematical content “$t\inT$” may be a complement to “$P_{t}$” and may therefore be associated as a supplementary mathematical content.

[0078] In another example, a tagged document may include metadata describing a symbolic variable P as "number of people". The metadata "number of people" may be extracted by the NER and relation extractor 204 as a metadata label for a "discrete variable" associated with the symbolic variable P. In yet another example, a tagged document may declare that the symbolic objective function P_{i}*X{i} "must be amplified", which may be extracted by the NER and relation extractor 204 as "maximizing" the objective direction associated with the symbolic expression P_{i}*X{i}. Thus, the NER and relation extractor may be trained to extract the metadata if present in the declaration, and may be additionally trained to understand the mathematical direction (or semantic meaning) of the metadata.

[0079] The NER and relation extractor 204 can be another trained neural network (e.g., based on a pre-trained BERT neural network) that has been trained using a training dataset that includes annotated segments (e.g., segments of text and / or mathematical content that have been annotated with labels of mathematical content entities, metadata tags, relationships, etc.). A neural network that can be used to implement the NER and relation extractor 204 can be trained in a manner similar to that described above with respect to the claim extractor 202.

[0080] Figure 3C An example of annotated training data that can be used to train the NER and relation extractor 204 is shown. Figure 3C As shown, markup declaration 504 has been annotated with a tag indicating a metadata entity and a tag indicating mathematical content. Figure 3D Another example is shown, where another markup statement 504 has been annotated with a label indicating a metadata entity and a label indicating mathematical content. In addition, there are nested labels within the mathematical labels indicating the meaning of the mathematical content (e.g., "LHS", "RHS", "Constraint", and "Direction" labels). For example, the annotations can be created using any suitable annotation software that supports nested annotations. Although not shown, the annotated training data can also include links indicating the relationship between the metadata and the mathematical content.

[0081] Figure 3E An exemplary neural network architecture that can be used to implement the NER and relation extractor 204 is shown. It should be understood that this is not intended to be limiting, and other neural network architectures may be suitable. For example, Giorgi et al. ("End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Models", arXiv:1912.13415, 2019) describes a suitable architecture for the NER and relation extractor 204.

[0082] The NER and relation extractor 204 can be thought of as having two interconnected branches, namely a NER branch capable of extracting metadata tags, semantic metadata, and mathematical content entities, and a relation extractor branch capable of extracting metadata-mathematical relations. The NER branch includes a pre-trained BERT-based neural network 210 (e.g., as described in Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” 2019) that processes tokenized declarations into embeddings, followed by a classifier subnetwork 212 and a softmax layer 214. The output of the softmax layer 214 is the NER output probabilities for predicting metadata tags, semantic metadata, and mathematical content entities.

[0083] The relation extractor branch receives the NER output probabilities and converts the predicted entity-tagged word IDs into embeddings using an embedding layer 216. The converted embeddings are processed in a cascade operation 218 together with the embeddings generated by the BERT-based neural network 210. This is followed by an attention pooling convolutional neural network (CNN) subnetwork 220 (e.g., as described in Wang et al., “Relation Classification via Multi-Level Attention CNNs,” 2016), a classifier subnetwork 222, and a softmax layer 224. The output of the softmax layer 224 is the relation extractor output probability, which is used to predict the metadata-mathematical relationship.

[0084] Thus, if at least one metadata tag or semantic metadata is extracted, the NER and relationship extractor 204 parses a single markup statement into one or more mathematical content entities (which may include a primary mathematical content entity and one or more auxiliary mathematical content entities), possible one or more metadata tags, possible one or more semantic metadata, and / or one or more metadata-mathematical relations. The NER and relationship extractor 204 may be trained to recognize a mathematical content entity in a markup statement as a supplementary mathematical content entity that is related to another mathematical content entity (which may be considered a primary mathematical content entity). The supplementary mathematical content entity may be related to another mathematical content entity within the same markup statement, and the NER and relationship extractor 204 may be trained to extract the relationship between the supplementary mathematical content entity and the primary mathematical content entity. For example, claim extractor 202 may be trained to extract labeled claims having multiple mathematical content entities (e.g., trained using annotated training data, wherein the annotated claims are further annotated to contain multiple mathematical content entities), and NER and relationship extractor 204 may be trained to identify each mathematical content entity in the labeled claim as a primary mathematical content entity or a supplementary mathematical content entity, and further identify a relationship between the primary mathematical content entity and each supplementary mathematical content entity (e.g., trained using annotated training data, wherein the mathematical content entities are annotated as supplementary or primary and further annotated with relationships between each other).

[0085] Each mathematical content entity is processed by a mathematical content parser 206. The mathematical content parser 206 receives mathematical content entities as input and extracts mathematical content and its relations. The output of the mathematical content parser 206 is a mathematical content parse tree, which is a tree data structure representing the components and relations of mathematical content entities. For example, a simple mathematical content entity (e.g., "a+b") consisting of two symbols with an operator between the two is parsed into a mathematical content parse tree by mathematical content parsing 206, wherein each symbol and operator form a node of the mathematical content parse tree, and the mathematical content entity forms the root of the mathematical content parse tree (e.g., "a", "+" and "b" will be the leaf nodes of "a+b"). For mathematical content entities including a target function, the mathematical content parsing 206 can output a more complex mathematical content parse tree, which has nodes corresponding to the symbols and corresponding indexes, operators, formulas and directions (e.g., maximization, minimization) constituting the target function. Each mathematical content entity in the one or more mathematical content entities extracted by the NER and relation extractor 204 is parsed by the mathematical content parser 206 to obtain corresponding one or more mathematical content parse trees.

[0086] The mathematical content parser 206 can be implemented using any suitable parser algorithm designed to parse mathematical expressions according to a defined mathematical grammar. For example, a context-independent grammar-based parser (e.g., generated using an ANTLR parser generator, such as described in Parr et al., "LL(*): The Foundation of the ANTLR Parser Generator", 2011) can be used as the mathematical content parser 206. It should be understood that since the mathematical content parser 206 expects to parse mathematical content entities in the context of the optimization problem, the type of mathematical expression that the mathematical content parser 206 expects to parse can be less than the pool of all possible mathematical expressions. This may help to simplify the implementation of the mathematical content parser 206 (e.g., the mathematical content parser 206 may only need to recognize certain types of mathematical grammars).

[0087] The mathematical content parse tree generated by the mathematical content parser 206, as well as any metadata tags, semantic metadata, and metadata-mathematical relationships generated by the NER and relationship extractor 204, are processed by the tree to definition converter 208 to generate a symbolic model. The symbolic model can be thought of as a data structure that represents the mathematical expression of the optimization problem. The symbolic model contains symbolic code, which is a structured way to represent mathematical content, but the symbolic code is not necessarily solver code (i.e., the symbolic code does not necessarily use a computational coding language that is compatible with the solver). The symbolic code can be converted to solver code by the model instantiator 300, as discussed further below. The tree to definition converter 208 processes each mathematical content parse tree by traversing each node, and, if any corresponding metadata exists, associates the corresponding metadata (e.g., metadata tags and semantic metadata) based on the metadata-mathematical relationship. The tree to definition converter 208 can then convert the relationships and any metadata represented by the mathematical content parse tree into symbolic code using predefined logic. For example, consider Figure 3F An example math content parse tree is shown.

[0088] exist Figure 3F , an exemplary mathematical content parse tree 230 for the symbolic expression “$\sum_{i\in F}v_{i}x_{i}\leq m$” is shown. The mathematical content parse tree 230 includes a leaf node containing the keyword “\sum”. The tree-to-definition converter 208 can follow the predefined rule that nearby leaf nodes contain symbols that describe indexes bound to sums. In this case, the node “i” is determined to be an index, and the node “\in” is detected as another keyword that connects the symbol “F” to the index. The tree-to-definition converter 208 can also detect other variables and parameters (such as “v”, “x”, and “m”) and any corresponding indexes (for example, following the predefined rule of identifying indexes between curly braces “{” and “}”). In this example, the mathematical content represented by the mathematical content parse tree 230 is associated with the metadata tag “constraint”, so the tree-to-definition converter 208 can use predefined logic to convert these identified fields into the code of the constraint. The tree-to-definition converter 208 may follow a predefined rule that the node "\leq" is a keyword identifying the direction of the constraint (in this case, the constraint direction is "less than or equal to"). As a result, the tree-to-definition converter 208 generates code for a constraint in which the sum on the left side is constrained to be less than or equal to the value on the right side, such as the code "ForAllSymConstraint" 232.

[0089] As described above, in some examples, a given declaration may contain only one or more mathematical content entities, without any metadata entities or metadata-mathematical relations. In such examples, only the mathematical content parse tree generated by mathematical content parsing 206 can be used by tree-to-definition converter 208 to generate symbolic code corresponding to the given declaration. After processing all mathematical content parse trees, the result is a symbolic model, which is a representation of the optimization problem, stripped of irrelevant information, such as markup language-specific requirements, human language tags, irrelevant delimiters / symbols, etc.

[0090] It should be noted that although the symbolic model does not contain semantic metadata (e.g., metadata containing human language text to help explain the optimization problem), the semantic metadata is still related to the corresponding mathematical content through the metadata-mathematical relationship. The optimization problem modeling system 100 can present the symbolic model or components of the symbolic model together with the relevant semantic metadata (e.g., through a user interface displayed on a user device) to enable human users (e.g., non-OR experts) to understand the symbolic model.

[0091] Figure 4 A simplified example of how the symbol model generator 200 processes a markup document is shown. It should be understood that for ease of understanding, only a portion of a markup document is shown and described below, but the symbol model generator 200 can process the entire markup document, not just a portion of the markup document.

[0092] The markup document 502 is received and processed by the declaration extractor 202. The declaration extractor 202 extracts the markup declaration 504, ignoring other text in the markup document 502. The markup declaration 504 is processed by the NER and relationship extractor 204, which extracts the mathematical content entity 506, the metadata tag 508, and the semantic metadata 510. The NER and relationship extractor 204 also extracts the metadata-mathematical relationship 512 (indicated by the curved dashed line), which associates the extracted metadata 508, 510 with the mathematical content entity 506. Each extracted mathematical content entity 506 is processed by the mathematical content parser 206 into a mathematical content parse tree 512 (for simplicity, the mathematical content parse tree 512 is only shown for one mathematical content entity 506). The mathematical content parse tree 512 is processed by the tree to definition converter 208 together with the metadata 508, 510 and the metadata-mathematical relationship 512 to generate a symbolic model. The generated symbolic model is associated with the extracted semantic metadata. In particular, each symbolic component (eg, symbolic variable, equation, parameter, etc.) of a symbolic model may be associated with corresponding semantic metadata that describes the meaning of the symbolic component using human language.

[0093] Optionally, the symbolic model may be saved, for example, stored in a symbolic model database 250, which may be local to the optimization modeling system 100 or may be an external database. The symbolic model may be stored in association with its corresponding semantic metadata.

[0094] The symbolic model database 250 may be searchable to enable a user to identify a desired symbolic model from among all the symbolic models stored in the symbolic model database 250. The optimization problem modeling system 100 may provide a UI (not shown) that receives a text string as a query, and may implement a search engine that identifies all stored symbolic models having associated semantic metadata that match the query, and returns the identified symbolic models as search results. In this way, by storing each symbolic model in association with its corresponding semantic metadata, a user may more easily search the stored symbolic models. In some examples, a user may retrieve a stored symbolic model for use as a template for developing a new symbolic model.

[0095] Optionally, the symbolic model or components of the symbolic model may be output for display by the optimization problem modeling system 100 or a display screen or other output device of a user device in communication with the optimization problem modeling system 100. This may enable a user to identify any errors in how the optimization problem is defined in the markup document (e.g., missing parameters may be readily identified).

[0096] Figure 5 An exemplary user interface (UI) 600 that can be displayed to a user is shown. UI 600 includes an input field 602 that displays a markup document. For example, the markup document can be manually input into input field 602 (e.g., by the user manually typing the markup document, or copying the content of the markup document into the input field), or can be uploaded from the memory of the user device. After being processed by at least declaration extractor 202 and NER and relationship extractor 204, the recognized symbol components (e.g., symbol parameters, variables, equations, constraints, etc.) can be displayed in symbol field 604.

[0097] The user may be provided with an option 606 to verify, accept, or edit (e.g., manually add, remove, or change constraints, change detected declaration types, etc.) each symbol component. It should be noted that the symbol field 604 displays each symbol component and the semantic metadata that has been associated with the symbol component. This can help non-OR experts understand the meaning of each symbol component, thereby improving understanding and / or accessibility, and enabling symbol models to be edited more efficiently.

[0098] The user may be provided with an option 608 to receive the recognized symbol components (after any edits by the user) and continue generating the symbol model. The user may also be provided with an option 610 to return to a previous step, such as uploading a different markup document.

[0099] It should be understood that UI 600 is illustrative only and is not intended to be limiting.

[0100] After the symbol model is generated by the symbol model generator 200, it can be processed into a model instance by the model instantiator 300. As described above, after the symbol model is generated, it can be stored in the symbol model database 250. The model instantiator 300 can retrieve the symbol model to be instantiated into a model instance from the symbol model database 250 at some later time.

[0101] As described above, the model instantiator 300 supports instantiation of model instances from a symbolic model. This means that the symbolic model can be decoupled from the data definition, so that different model instances can be generated using different format combinations of the markup document and the data definition (e.g., the format of the data definition does not need to depend on the markup language used to markup the document, and vice versa). Some existing methods for modeling optimization problems require the markup document to use mathematical formulas that correspond exactly to the data definition, and any changes to the data definition (e.g., as new data is collected, or new parameters are introduced for the optimization problem) require the expertise of an OR expert to develop a new model instance. This can be costly and time consuming, and can be avoided through the examples of the present invention.

[0102] The model instantiator 300 is capable of generating model instances suitable for any suitable solver 10. The user may only need to indicate the desired solver configuration without creating a markup document or data definition in a specific format required by the solver 10. This may also enable different solvers 10 to be used to solve optimization problems defined by the same markup document and the same data definition.

[0103] In general, the model instantiator 300 may perform the conversion of a data definition (which may be received as a single data definition file, or each data definition may be a corresponding data definition file) into a format suitable for the configuration of the selected solver 10 (e.g., the format of the definition collection and parameters according to the selected solver API). The model instantiator 300 also defines variables and all possible types according to the selected solver configuration, processes symbolic expressions in the symbolic model to define constraints and objectives according to the selected solver configuration, and generates a model instance that can be received by the solver 10. The generated model instance can be directly transmitted to the solver 10 to generate a solver result for the optimization problem and / or the model instance can be saved (e.g., saved as a .mps file in an internal or external memory) for future use.

[0104] Figure 6 is a block diagram illustrating an exemplary implementation of the model instantiator 300. Although the model instantiator 300 is illustrated with certain functional blocks, it should be understood that this is not intended to be limiting.

[0105] The model instantiator 300 receives as input a symbolic model and one or more data definitions. Optionally, a solver configuration (e.g., an indication of a solver API for generating a model instance) may also be provided as input to the model instantiator 300. If a solver configuration is not provided to the model instantiator 300, a default solver configuration (e.g., a default solver API) may be used.

[0106] The symbolic model and data definitions are processed into a data definition map by a data definition mapper 302. The data definition map contains the data definitions and maps each data definition to a corresponding symbolic parameter of the symbolic model. An instantiation code generator 304 processes the symbolic model, the data definition map, and an optional solver configuration to generate a model instance in the form of computer readable code that can be received as input by the solver 10 (i.e., in the form of solver code).

[0107] The data definitions may be provided by a user (e.g., uploaded from a user device). Each data definition provides a numerical value for a corresponding set of symbols or symbol parameters in a symbol model. The data definitions may be provided as one or more software files (e.g., each file containing data definitions for corresponding parameters) or in any suitable format (e.g., received as a data stream, or manually entered). A common format for data definitions is a spreadsheet file, in which the data is presented in a tabular form (which is a common method for representing two-dimensional parameters). In examples where the data definitions are presented in a tabular form, the parameters may be identified based on the file name (e.g., based on the file name corresponding to the symbol or semantics of the parameter), and the column and row headers may correspond to the index.

[0108] Optionally, a solver configuration may be provided as user input that selects which solver API to use and / or which solver 10 to use. For example, the solver configuration may provide an indication of which specific solver API to use (e.g., from available solver APIs, such as Pyomo API, PuLP API, etc.), as well as an indication of the solver type (e.g., from possible solver types, such as OR-COIN, branch and bound, Cplex, etc.). The solver configuration may also indicate whether the model instance should be saved and / or exported to a specific file format suitable for the optimization model (e.g., .mps file, .rew file, .dua file, etc.).

[0109] The symbol model and the data definition are provided as inputs to the data definition mapper 302. The data definition mapper 302 generates data definition maps. Each data definition map maps a corresponding data definition to a corresponding symbol parameter defined in the symbol code of the symbol model. For example, the data definition mapper 302 may use a rule-based approach to identify symbol parameters in the symbol model (e.g., each symbol parameter may be so labeled in the symbol model) and match each symbol parameter to a data definition file having the same file name.

[0110] In some examples, a user can be notified of any symbol parameters that cannot be mapped to a data definition or any data definitions that cannot be mapped to a symbol parameter. For example, a UI can be provided via a user device, wherein a notification can be displayed for any differences between the data definition and the symbol parameters of the symbol model. This can provide feedback to the user to enable the user to collect or provide one or more missing data definitions, or to enable the user to edit the symbol model appropriately.

[0111] For example, if there is no data definition for a given symbolic parameter, the UI can display the symbolic parameter. The symbolic parameter can be displayed with associated semantic metadata to enable the user to understand the meaning of the symbolic parameter. The symbolic parameter can also be displayed in the context of a mathematical equation, where the symbolic parameter appears in the symbolic model. Optionally, a blank data definition template (e.g., a blank spreadsheet file) with appropriate indexes, headers, etc. can be automatically generated, which the user can use to collect the missing data definitions.

[0112] The data definition mapping, the symbolic model, and the optional solver configuration are received by the instantiation code generator 304. The instantiation code generator 304 uses the data definition mapping to instantiate and load the numerical data values ​​of the symbolic parameters defined in the symbolic model according to the format defined by the solver API indicated by the solver configuration (or according to a default format if no solver configuration is provided). The instantiation code generator 304 can use classical post-processing methods to generate code for the model instance, or can use machine learning methods (e.g., using an encoder-decoder model) to generate the model instance.

[0113] For example, if the instantiated code generator 304 uses a classical post-processing approach, the instantiated code generator 304 can process the symbolic model using defined rules (e.g., based on a known symbolic code structure) to convert statements in the symbolic code into statements in the solver code (according to the format defined by the solver API). The instantiated code generator 304 can use a data definition mapping to load numerical data values ​​from the data definition into the solver code (e.g., if the data definition mapping maps the symbolic parameter a to a spreadsheet containing the values ​​"1, 2, 3", the generated solver code can be "a = [1, 2, 3]").

[0114] In another example, if the instantiated code generator 304 uses a machine learning approach, the instantiated code generator 304 can be a trained encoder-decoder model. The encoder-decoder model can be trained using symbolic model-model instance pairs. Each symbolic model-model instance pair includes a symbolic model (and associated data definition mapping) paired with a target model instance. The encoder-decoder model can then be trained to encode the symbolic model (and associated data definition mapping) into a potential representation, which is then decoded into a target model instance. In this way, the encoder-decoder model can be trained in a manner similar to a translator in order to "translate" the symbolic model into a model instance.

[0115] The model instance may then be stored or provided directly to the solver 10 to generate solver results for the optimization problem.

[0116] Figure 7 is a flow chart illustrating an exemplary method 700 for generating a symbolic model from a markup document. The method 700 may be performed by a computing system (e.g., computing system 400) that is configured to execute instructions to implement the functionality of the symbolic model generator 200 as described above. In some examples, the method 700 may be performed by a computing system that is configured to execute instructions to implement the optimization problem modeling system 100 (which may include the symbolic model generator 200 as a subsystem).

[0117] At 702, a marked-up document representing an optimization problem is received (eg, from a user device). The marked-up document may be an unstructured multimodal document including both human language content and mathematical content.

[0118] At 704, the tagged document is analyzed to extract one or more tagged claims from the tagged document. As described above, a trained neural network (e.g., an NLP-based neural network that has been trained on annotated training data sets) can be used to parse the tagged document to extract the one or more tagged claims. For example, claim extractor 202 can be used to perform step 704.

[0119] At 706, each markup statement is processed to extract at least one mathematical content entity. If the statement contains human language content, any metadata entities in the statement and any relationships between the extracted metadata entities and the extracted mathematical content entities are also extracted. As described above, another trained neural network (e.g., a neural network based on NLP, such as Figure 3E The neural network architecture shown, which has been trained on an annotated training dataset, can be used to process the labeled statement to extract one or more mathematical content entities as well as any metadata entities (e.g., one or more metadata tags and / or semantic metadata) and any metadata-mathematical relations (depending on the content of the statement). For example, the NER and relation extractor 204 can be used to perform step 706.

[0120] In 708, each math content entity is processed to generate a corresponding math content parse tree. The math content parse tree is a tree data structure that represents the components (e.g., symbols and operators) of the math content entity and the relationships between the components. For example, the math content parser 206 (which can be generated using an ANTLR parser generator) can be used to perform step 708.

[0121] At 710, each math content parse tree is converted to a symbolic code to generate a symbolic model. If there are any metadata entities related to the math content entity represented by the math content parse tree, the metadata entity is also used to generate the symbolic code from the math content parse tree. For example, the tree to definition converter 208 can process the math content parse tree and any related metadata (identified by the metadata-mathematical relationship) to generate the symbolic code.

[0122] After processing all the mathematical content parse trees, a symbolic model representing the optimization problem is generated.

[0123] The symbol model is output along with the associated semantic metadata at 712. Each piece of metadata may be associated with a corresponding symbol content of the symbol model, as defined by a metadata-mathematical relationship.

[0124] The symbol model and associated semantic metadata may be output for storage in a memory (e.g., in the symbol model database 250) and / or may be output for instantiation as a model instance (e.g., by the model instantiator 300). Optionally, the symbol model and associated semantic metadata may also be output for display to a user (e.g., via a UI on a user device) to enable the user to view and / or edit the symbol model.

[0125] Figure 88 is a flow chart illustrating an exemplary method 800 for instantiating a model instance from a symbolic model. The method 800 may be performed by a computing system (e.g., computing system 400) that is configured to execute instructions to implement the functionality of the model instantiator 300 as described above. In some examples, the method 800 may be performed by a computing system that is configured to execute instructions to implement the optimization problem modeling system 100 (which may include the model instantiator 300 as a subsystem).

[0126] In 802 , a symbolic model representing the optimization problem is obtained. For example, the symbolic model can be obtained directly from the symbolic model generator 200 , or can be obtained from the symbolic model database 250 .

[0127] At 804, one or more data definitions are obtained (eg, received from a user device). The data definitions represent values ​​for symbol parameters of a symbol model. The data definitions may be obtained in any suitable format (eg, a spreadsheet file).

[0128] Optionally, in 806, a solver configuration may be obtained (e.g., received from a user device). The solver configuration may indicate a solver API to be used, or may indicate a solver 10 to be used (thus implying a solver API to be used). In some examples, the solver configuration may not be obtained, and a default solver configuration (e.g., a default solver API) may be used.

[0129] At 808, one or more data definition mappings are generated that map each data definition to a corresponding symbol parameter in the symbol model. For example, data definition mapper 302 may generate the data definition mappings using predefined rules (e.g., mapping each data definition to a corresponding symbol parameter by matching the file name of each data definition to semantic metadata associated with each symbol parameter).

[0130] Optionally, if there are any symbol parameters that cannot be mapped to the data definition, or vice versa, a notification can be generated and output to the user device to inform the user of the discrepancy. This can enable the user to provide the missing data definition and / or edit the symbol model appropriately.

[0131] At 810, a model instance is instantiated from the symbolic model using the data definition mapping, the data definition, and the optional solver configuration. For example, the instantiation code generator 304 may generate the model instance using a classical post-processing approach or a machine learning-based approach.

[0132] At 812 , the generated model instance is output to be solved by the solver 10 .

[0133] Methods 700 and 800 may be performed together (e.g., to generate a model instance from a markup document), or may be performed separately (e.g., to generate a symbolic model from a markup document, and to generate a model instance from the symbolic model, respectively). Methods 700 and 800 may be performed by the same computing system or by different computing systems.

[0134] Examples of the present invention may enable non-OR experts to generate symbolic models and model instances from unstructured multimodal labeled documents describing optimization problems. This may help improve the accessibility of existing solvers to non-OR experts.

[0135] This invention describes a hierarchical parsing method for generating a symbolic model from a marked-up document. In other examples, a single neural network can be used to perform the parsing. For example, instead of having a claim extractor as separate components for NER and a relation extractor, a single machine learning-based component can be trained to perform claim extraction as well as named entity recognition and relation extraction.

[0136] The present invention describes methods and systems in which the conversion of markup documents to symbolic models and the conversion of symbolic models to model instances are decoupled from each other. This may help with troubleshooting, model revisions, etc. Decoupling symbolic models from model instances allows model instantiation to be more flexible and adaptable to different data definition formats and / or different solver configurations. In addition, symbolic models can be saved with associated semantic metadata and retrieved later by querying the associated semantic metadata. Saved symbolic models can also serve as templates for generating new symbolic models.

[0137] In some examples, symbolic models may not be stored in a database (or other memory), but may be directly instantiated as model instances without being saved for future use.

[0138] Examples of the present invention may be provided as a cloud-based service (e.g., accessible to users through a web-based portal, through a mobile application, through a software package, etc.). The present invention may be applicable to optimization problems in any field (i.e., not limited to a particular industry or application).

[0139] In some examples, a UI may be provided to enable a user to provide input (e.g., enter a markup document, edit parameters of a symbol model, upload a data definition, etc.), retrieve a symbol model from a database (e.g., enter a query to search the database), review a symbol model, etc. In some examples, a user may use the UI to search a symbol model database for a symbol model that may be used as a template for developing a new symbol model. The UI may enable a user to view symbol parameters and equations of a symbol model, as well as associated semantic metadata, to help the user understand and edit the symbol model.

[0140] Although examples have been described in which the input marked-up document is an unstructured multimodal document, it should be understood that, for example, the present invention may also be applicable to situations where the marked-up document is structured (e.g., follows specific rules for defining an optimization problem), or contains only mathematical content.

[0141] Although the present invention describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be omitted or modified as appropriate. One or more steps may be performed in an order other than the order described as appropriate.

[0142] Although the present invention has been described, at least in part, in terms of methods, it will be understood by those skilled in the art that the present invention also relates to various components for performing at least some aspects and features of the described methods by hardware components, software, or any combination of the two. Accordingly, the technical solution of the present invention can be embodied in the form of a software product. Suitable software products can be stored in a pre-recorded storage device or other similar non-volatile or non-transient computer-readable medium, such as a DVD, CD-ROM, USB flash drive, removable hard disk or other storage medium. The software product includes instructions tangibly stored therein, which enable a processing device (e.g., a personal computer, a server, or a network device) to perform examples of the methods disclosed herein. Machine executable instructions can be in the form of code sequences, configuration information, or other data, which, when executed, enable a machine (e.g., a processor or other processing device) to perform the steps in the methods provided by the examples of the present invention.

[0143] The present invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative and not restrictive in all respects. Features selected from one or more of the above-described embodiments may be combined to create optional embodiments not explicitly described, and features suitable for such combinations may be understood within the scope of the present invention.

[0144] All values ​​and subranges within the disclosed range are also disclosed. In addition, although the systems, devices, and processes disclosed and shown herein may include a specific number of elements / components, the systems, devices, and components may be modified to include more or fewer elements / components in such elements / components. For example, although any disclosed element / component may be cited as a singular, the embodiments disclosed herein may be modified to include a plurality of such elements / components. The subject matter described herein is intended to cover and encompass all suitable technical changes.

Claims

1. A computing system, It is characterized in that include: a processing unit for executing computer readable instructions to cause the system to: receiving a marked-up document representing an optimization problem, the marked-up document comprising human language content and mathematical content; The markup document is parsed into a symbolic model containing only symbolic codes representing the optimization problem by: extracting at least one markup declaration from the markup document; For the at least one markup statement, extracting at least one corresponding mathematical content span, and extracting any corresponding metadata entities and any relationships between the at least one corresponding mathematical content span and any corresponding metadata entities; For the at least one mathematical content span, generating a corresponding mathematical content parse tree; converting the corresponding math content parse tree into a symbolic code using any relationship between the at least one corresponding math content span and any corresponding metadata entity, the symbolic code forming the symbolic model; The symbolic model associated with the semantic metadata including the at least one corresponding metadata entity is output.

2. The computing system according to claim 1, It is characterized in that The processing unit is further configured to execute the instructions so that the system: implementing a declaration extractor using a first trained neural network to extract the at least one markup declaration from the markup document; A named entity recognition and relationship extractor is implemented using the second trained neural network to extract the at least one corresponding mathematical content span, any corresponding metadata entities, and any relationships between the at least one corresponding mathematical content span and any corresponding metadata entities.

3. The computing system according to claim 2, It is characterized in that The first trained neural network and the second trained neural network are respectively based on a pre-trained natural language processing neural network.

4. A computing system according to any one of claims 1 to 3, It is characterized in that The processing unit is further configured to execute the instructions so that the system: A user interface is provided for receiving the markup document and for displaying at least one symbolic parameter of the symbolic model, the at least one symbolic parameter corresponding to the at least one mathematical content span, the at least one symbolic parameter being displayed together with any associated corresponding metadata entities.

5. The computing system according to claim 4, It is characterized in that The processing unit is further configured to execute the instructions so that the system: Input is received to edit the at least one symbol parameter.

6. A computing system according to any one of claims 1 to 5, It is characterized in that The symbolic model associated with the semantic metadata is stored in a symbolic model database.

7. A computing system according to any one of claims 1 to 6, It is characterized in that The processing unit is further configured to execute the instructions so that the system: obtaining at least one data definition; generating at least one data definition mapping, the at least one data definition mapping mapping the at least one data definition to at least one symbol parameter of the symbol model; processing the symbolic model using the at least one data definition mapping and the at least one data definition to generate a model instance; Outputs the model instance to be solved by the solver.

8. The computing system according to claim 7, It is characterized in that The processing unit is further configured to execute the instructions so that the system: Get solver configuration; The model instance is generated according to the solver configuration.

9. A computing system according to claim 7 or 8, It is characterized in that The processing unit is further configured to execute the instructions to cause the system to: identify a difference between the at least one data definition and the at least one symbol parameter; A notification indicating the difference is output.

10. A computing system according to any one of claims 7 to 9, It is characterized in that The symbol model is stored in a database, and the processing unit is further configured to execute the instructions to cause the system to retrieve the symbol model from the database in response to a query.

11. A method, It is characterized in that include: receiving a marked-up document representing an optimization problem, the marked-up document comprising human language content and mathematical content; The markup document is parsed into a symbolic model containing only symbolic codes representing the optimization problem by: extracting at least one markup declaration from the markup document; For the at least one markup statement, extracting at least one corresponding mathematical content span, and extracting any corresponding metadata entities and any relationships between the at least one corresponding mathematical content span and any corresponding metadata entities; For the at least one mathematical content span, generating a corresponding mathematical content parse tree; converting the corresponding math content parse tree into a symbolic code using any relationship between the at least one corresponding math content span and any corresponding metadata entity, the symbolic code forming the symbolic model; The symbolic model associated with the semantic metadata including the at least one corresponding metadata entity is output.

12. The method according to claim 11, It is characterized in that Also includes: implementing a declaration extractor using a first trained neural network to extract the at least one markup declaration from the markup document; A named entity recognition and relationship extractor is implemented using the second trained neural network to extract the at least one corresponding mathematical content span, any corresponding metadata entities, and any relationships between the at least one corresponding mathematical content span and any corresponding metadata entities.

13. The method according to claim 12, It is characterized in that The first trained neural network and the second trained neural network are respectively based on a pre-trained natural language processing neural network.

14. The method according to any one of claims 11 to 13, It is characterized in that Also includes: A user interface is provided for receiving the markup document and for displaying at least one symbolic parameter of the symbolic model, the at least one symbolic parameter corresponding to the at least one mathematical content span, the at least one symbolic parameter being displayed together with any associated corresponding metadata entities.

15. The method according to claim 14, It is characterized in that Also includes: Input is received to edit the at least one symbol parameter.

16. The method according to any one of claims 11 to 15, It is characterized in that The symbolic model associated with the semantic metadata is stored in a symbolic model database.

17. The method according to any one of claims 11 to 16, It is characterized in that Also includes: obtaining at least one data definition; generating at least one data definition mapping, the at least one data definition mapping mapping the at least one data definition to at least one symbol parameter of the symbol model; processing the symbolic model using the at least one data definition mapping and the at least one data definition to generate a model instance; Outputs the model instance to be solved by the solver.

18. The method according to claim 17, It is characterized in that Also includes: Get solver configuration; The model instance is generated according to the solver configuration.

19. The method according to claim 17 or 18, It is characterized in that Also includes: identifying a difference between the at least one data definition and the at least one symbol parameter; A notification indicating the difference is output.

20. The method according to any one of claims 17 to 19, It is characterized in that The symbol model is stored in a database, and the method further comprises retrieving the symbol model from the database in response to a query.

21. A non-transitory computer readable medium, It is characterized in that Instructions are stored therein, wherein the instructions can be executed by a processing unit of a computing system to enable the computing system to perform the method according to any one of claims 11 to 20.

Images