Interactive constraint satisfaction with generative language models

Abstract

The disclosed concepts relate to interactive constraint solving with generative language models. For example, a generative language model can be prompted to generate a constraint data structure expressed in natural language that represents user preferences. The constraint data structure can be parsed to extract constraint parameters that can be programmatically solved by a constraint solver. The generative language model can also be prompted to generate constraint checking code that can be invoked by the constraint solver.

Description

Background Technology

[0001] Traditional constraint satisfaction algorithms can identify solutions to very complex problems with numerous constraints on many variables. However, these algorithms typically require constraints to be fully specified beforehand. Furthermore, traditional constraint satisfaction software requires constraints to be expressed programmatically in a software-specific data format that is not easily understood by most users. Summary of the Invention

[0002] This summary is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0003] This specification generally relates to constraint satisfaction in relation to generative language models. One example includes a computer-implemented method that may include: receiving natural language input from a user, the natural language input specifying the user's preferences in natural language. The method may further include: generating a constraint management cue word for the generative language model, the constraint management cue word being based on the natural language input and including an instruction requesting the generative language model to generate a constraint data structure representing the preferences according to a specified constraint data format. The method may further include: inputting the constraint management cue word into the generative language model. The method may further include: receiving from the generative language model a constraint data structure generated by the generative language model, the constraint data structure being in the specified constraint data format. The method may further include: parsing the constraint data structure generated by the generative language model to extract constraint parameters. The method may further include: processing the constraint parameters using a constraint solver to identify candidate solutions that satisfy at least some of the constraint parameters. The method may further include: outputting candidate solutions to a user. The method may further include: updating a specific data source with the accepted solution in response to user input identifying an accepted solution from the candidate solutions.

[0004] Another example relates to a system including a hardware processing unit and storage resources storing computer-readable instructions. When executed by the hardware processing unit, the computer-readable instructions can cause the system to: receive natural language input from a user, the natural language input specifying the user's preferences in natural language. The computer-readable instructions can also cause the system to: generate constraint management prompts for a generative language model, the constraint management prompts being based on the natural language input and including instructions that request the generative language model to generate a constraint data structure with constraint parameters representing the user's preferences. The computer-readable instructions can also cause the system to: generate code generation prompts for the generative language model, the code generation prompts instructing the generative language model to generate constraint checking code that checks whether possible solutions satisfy the constraint parameters. The computer-readable instructions can also cause the system to: utilize a constraint solver to execute the constraint checking code to identify candidate solutions that satisfy at least some of the constraint parameters. The computer-readable instructions can also cause the system to: output candidate solutions to a user. The computer-readable instructions can also cause the system to: update a specific data source with the accepted solution in response to user input that identifies an accepted solution from the candidate solutions.

[0005] Another example includes a computer-readable storage medium storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform an action. The action may include: receiving natural language input from a user, the natural language input specifying the user's preferences in natural language. The action may also include: generating a constraint management cue word for a generative language model, the constraint management cue word including preferences and instructions that request the generative language model to generate a constraint data structure representing preferences according to a specified constraint data format. The action may also include: inputting the constraint management cue word into the generative language model. The action may also include: receiving a constraint data structure generated by the generative language model from the generative language model, the constraint data structure being in the specified constraint data format. The action may also include: parsing the constraint data structure generated by the generative language model to extract constraint parameters. The action may also include: processing the constraint parameters using a constraint solver to identify candidate solutions that satisfy at least some of the constraint parameters. The action may also include: outputting candidate solutions to a user. The action may also include: updating a specific data source with the accepted solution in response to user input identifying an accepted solution from the candidate solutions.

[0006] The examples listed above are intended to provide a quick reference to help the reader and are not intended to limit the scope of the ideas described in this article. Attached Figure Description

[0007] Detailed embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost number(s) of the reference numerals identify the drawing in which that reference numeral first appears. In different instances in the specification and drawings, the use of similar reference numerals may indicate similar or identical items.

[0008] Figure 1 Examples of generative language models based on some implementations of the proposed concept are shown.

[0009] Figure 2 Examples of user interactions with interactive constraints satisfying intelligent agents are shown, based on some implementations of the proposed concept.

[0010] Figure 3A , Figure 3B , Figure 3C and Figure 3D Examples of constraint management prompt word templates based on some implementations of the proposed concept are shown.

[0011] Figure 4A and Figure 4B Examples of data inspection prompt word templates based on some implementations of the proposed concept are shown.

[0012] Figure 5A and Figure 5B Examples of code-generated prompt word templates based on some implementations of the proposed concept are shown.

[0013] Figure 6A and Figure 6B Examples of suggested explanatory prompt templates based on some implementation methods of the proposed concept are shown.

[0014] Figure 7A , Figure 7B and Figure 7C Example graphical user interfaces are shown, illustrating some implementations of the proposed concept.

[0015] Figure 8 Experimental results obtained using some implementations of the proposed concept are shown.

[0016] Figure 9 Example systems are shown that implement some of the proposed ideas.

[0017] Figure 10 Examples of methods or techniques for constraint satisfaction using generative language models are shown, based on some implementations of the proposed concept. Detailed Implementation

[0018] Overview As mentioned above, constraint satisfaction algorithms can identify solutions to complex constraint satisfaction problems involving numerous constraints on many variables. However, most users cannot easily express constraints in a suitable format that can be used with traditional constraint solvers. Furthermore, constraints such as user preferences may change over time or in different contexts (e.g., at home versus in the office), and in fact, users may not even be fully aware of all their preferences when initially attempting to complete a task. However, traditional constraint solvers are not good at directly extracting user preferences from the user.

[0019] In recent years, generative language models have significantly reduced the workload for users performing complex language tasks, such as generating captions for images or summarizing large documents. Generative language models are also able to understand user preferences expressed in natural language. However, despite the tremendous progress in generative language modeling technology, it may still lack sufficient capability in performing complex constraint-based reasoning on data.

[0020] The disclosed implementation leverages the strengths of both generative language models and constraint solving algorithms through a hybrid approach that integrates both. For example, the implementation could employ a generative language model to receive user preferences in natural language format and generate constraint data structures that formally represent those preferences in a specific data structure format. These constraint data structures can then be parsed to extract constraint parameters, which are input to a constraint solver to identify candidate solutions that match the user preferences.

[0021] The disclosed implementation can also employ a generative language model to generate constraint checking code, which can be called by the constraint solver to check whether a given candidate solution satisfies given constraint parameters. The generative language model can also be used to guide user interactions; for example, it can select an action to take in response to user input. Actions can include adding or deleting constraints, changing constraints, suggesting candidate solutions to the user, sending messages to the user, etc. In this way, a generative language model can be used to control the procedural flow of interactive constraint satisfaction of an agent, as described in more detail below.

[0022] Machine Learning Overview There are various types of machine learning frameworks that can be trained to perform a given task. Support vector machines, decision trees, and neural networks are just a few examples of machine learning frameworks already used in a wide variety of applications, such as image processing and natural language processing. Some machine learning frameworks, such as neural networks, use layers of nodes that perform specific operations.

[0023] In a neural network, nodes are connected to each other via one or more edges. A neural network may include an input layer, an output layer, and one or more intermediate layers. Each node can process its respective input according to a predefined function and provide output to subsequent layers or, in some cases, to previous layers. The input to a given node can be multiplied by the corresponding weight value of the edge between the input and the node. Furthermore, nodes may have individual bias values ​​that are also used to produce the output. Various training procedures can be applied to learn the edge weights and / or bias values. When used without embellishment, the term "parameter" in this document refers to learnable values, such as edge weights and bias values, that can be learned by training a machine learning model, such as a neural network.

[0024] Neural network architectures can have different layers that perform different specific functions. For example, one or more layers of nodes can work together to perform specific operations such as pooling, encoding, or convolution. For the purposes of this document, the term "layer" refers to a set of nodes that share inputs and outputs, such as those going to or from external sources or other layers in the network. The term "operation" refers to a function that can be performed by one or more layers of nodes. The term "model architecture" refers to the overall architecture of a layered model, including the number of layers, the connectivity of the layers, and the types of operations performed by each layer. The term "neural network architecture" refers to the model structure of a neural network. The terms "trained model" and / or "tuned model" refer to the model structure and the parameters of a trained or tuned model structure. Note that two trained models can share the same model structure but have different values ​​for their parameters, for example, if the two models are trained on different training data, or if there is an underlying stochastic process during training.

[0025] There are many machine learning tasks for which training data is relatively scarce. A widely used approach to train a model using limited task-specific training data for a particular task involves "transfer learning." In transfer learning, the model is first pre-trained on another task on which effective training data is available, and then fine-tuned for the specific task using task-specific training data.

[0026] As used in this paper, the term "pre-training" refers to training a model on a set of pre-training data, allowing these parameters to be further fine-tuned to adapt the model to one or more specific tasks. In some cases, pre-training can employ a self-supervised learning process on unlabeled pre-training data, where "self-supervised" learning refers to learning from the structure of the pre-training samples themselves, without relying on explicit (e.g., manually provided) labels. This paper refers to the subsequent adjustment of the pre-trained model parameters as "tuning." Tuning can be performed on one or more tasks using supervised learning with explicitly labeled training data; in some cases, the task used for tuning differs from the pre-training task.

[0027] the term For the purposes of this document, the term "language model" refers to any type of automated intelligent agent programmed to understand natural language and / or to communicate via natural language. For example, a language model can be implemented as a neural network, such as decoder-based generative language models like ChatGPT, BLOOM, PaLM, and / or LLaMA or variants thereof, long short-term memory models, etc. As used herein, the term "generative model" refers to a machine learning model used to generate new content. A generative model can be trained to predict items in a sequence of training data. When employed in inference mode, the output of a generative model can include a new sequence of items generated by the model. A "generative language model" is a model trained from one or more natural language training data sources to predict a sequence of output lexical units given one or more input lexical units. A generative language model can generate new sequences of text given some input prompts (e.g., a query that may have some additional context). In some cases, a generative language model can be multimodal; for example, in addition to text input and / or output, the model may be able to use images, audio, or other modalities as input and / or generate images, audio, or other modalities as output.

[0028] As used herein, the term "cue word" refers to the input text provided to a generative language model, which uses this input text to generate output text. Cue words can include queries, such as requests for information from the generative language model. Cue words can also include context or additional information that the generative language model uses to respond to queries. In some cases, cue words can include one or more examples from the generative language model as context (e.g., "few-sample cue words"), and the generative language model can be conditionalized to generate more accurate responses than it would produce without examples. As used herein, the term "in-context learning" refers to learning by a generative model from examples input to the model at inference time, where the examples enable the generative model to learn without performing explicit training, such as updating model parameters without using supervised, unsupervised, or semi-supervised learning.

[0029] The term "constraint" refers to any criterion used to solve a problem. Constraints can include, for example, constraint parameters for a meeting, which can specify attendees, locations, times, etc. Another type of constraint parameter is the constraint's "priority," which refers to how important the constraint is. For example, a constraint can be a high-priority constraint, such as a mandatory or "hard" constraint. A constraint can also be a low-priority constraint, such as a flexible or "soft" constraint. A constraint solver is a module that uses constraint satisfaction algorithms to solve constraints. A constraint solver can identify one or more candidate solutions for a given problem. In some cases, a constraint solver can rank candidate solutions by weighting them based on their respective priorities. A "data source" is any mechanism that allows data to be stored, updated, retrieved, or deleted. For example, a data source can be a database, spreadsheet, word processing document, etc., implemented in memory or storage devices.

[0030] The term "machine learning model" refers to any model in a broad class of models that can be trained to predict properties of input data or generate new data given some inputs. For example, a machine learning model can be a neural network, support vector machine, decision tree, clustering algorithm, etc. In some cases, machine learning models can be trained using labeled training data, reward functions, or other mechanisms, and in others, they can learn by analyzing data without explicit labels or rewards. The term "user-specific model" refers to a model having at least one component that is at least partially trained or constructed for a specific user. Therefore, this term includes models that have been fully trained for a specific user, models initialized with multi-user data and fine-tuned for a specific user, and models with general components trained for multiple users and one or more components trained or fine-tuned for a specific user. Similarly, the term "application-specific model" refers to a model having at least one component that is at least partially trained or constructed for a specific application.

[0031] The term "pruning" refers to removing parts of a machine learning model while retaining others. For example, a large machine learning model can be pruned into a smaller model for a specific task by retaining weights and / or nodes that significantly contribute to the model's ability to perform that specific task, while removing other weights or nodes that do not significantly contribute to the model's ability to perform that specific task. A large machine learning model can be distilled into a smaller model for that specific task by training a smaller machine learning model to approximate its output distribution on a task-specific dataset.

[0032] Example of a decoder-based language model Figure 1An exemplary generative language model 100 is shown that can be employed using the disclosed implementation. The generative language model 100 is an example of a machine learning model that can be used to perform one or more natural language processing tasks involving the generation of text, as discussed further below. For the purposes of this document, the term "natural language" means the language that humans typically use for writing or conversation.

[0033] Generative language model 100 can receive input text 110, such as prompt words from a user or a computer-based agent. For example, the input text can include words, sentences, phrases, or other language representations. The input text can be segmented into lexical units and mapped to lexical and positional embeddings 101 representing the input text. Lexical embeddings can be represented in a vector space, where semantically and / or syntactically similar embeddings are relatively close to each other, and less semantically or syntactically similar embeddings are relatively far apart. Positional embeddings represent the sequential position of each lexical unit relative to other lexical units from the input text.

[0034] Lexical and positional embeddings 101 are processed in one or more decoder blocks 102. Each decoder block implements masked multi-head self-attention 103, which is a mechanism that correlates different positions of lexical units within the input text to compute similarity between those units. Each lexical embedding is represented as a weighted sum of other lexical units in the input text. Attention is applied only to values ​​that have already been decoded, and future values ​​are masked. Layer normalization 104 normalizes the features to a mean of 0 and a variance of 1, resulting in smooth gradients. Feedforward layers 105 transform these features into representations suitable for the next decoding iteration, after which another layer normalization 106 is applied. Multiple instances of decoder blocks can sequentially operate on the input text, with each subsequent decoder block operating on the output of the previous decoder block. After the final decoder block, a text prediction layer 107 can predict the next word in the sequence, which is output as output text 120 in response to input text 110 and also fed back into the language model. The output text can be a newly generated response to the prompt words provided as input text to the generative language model.

[0035] Generative language model 100 can be trained on large and diverse document corpora using techniques such as next-word prediction or masked language modeling. For example, text prediction layer 107 can predict the next word in a given document, and the parameters of decoder block 102 and / or text prediction layer can be adjusted when the predicted word is incorrect. In some cases, the generative language model can be pre-trained on a large document corpus and then tuned to a specific use case. For example, reinforcement learning techniques such as reinforcement learning from human feedback (“RLHF”) can be used to tune the pre-trained generative language model.

[0036] Schedule Example The techniques described in this paper can be used to satisfy a wide range of interactive constraints using generative language models. A specific implementation is described below, where constraints are based on user preferences expressed in the context of scheduling a meeting.

[0037] Figure 2 User 202 and dialogue history 204 are shown, where the dialogue history involves interaction with the interactive constraint-fulfilling agent 210. The interactive constraint-fulfilling agent uses a generative language model 100 to interactively solve constraints expressed by the user in natural language. The interactive constraint-fulfilling agent includes a constraint manager 211, a constraint generator 217, and a constraint solver 220. The constraint manager can perform actions such as adding constraints 212, deleting constraints 213, changing priorities 214, generating suggestions 215, and / or sending messages to the user 216 based on communications from user 202 and outputs from the generative language model. The constraint generator 217 includes a data inspector 218 and a code generator 219. The data inspector can use a generative language model to determine whether the available data sources allow a given constraint to be inspected. The code generator can use a generative language model to generate constraint inspection code based on the constraints provided by the constraint manager and provide the constraint inspection code to the constraint solver 220. Communication 221 can be sent to the user to convey messages from the constraint manager or candidate suggestions provided by the constraint solver.

[0038] As described in more detail below, constraint solver 220 is used to generate new time suggestions that satisfy the constraint parameters generated by generative language model 100. The constraint solver performs the mathematical and / or logical reasoning required to find suitable times. Each scheduling preference expressed by user 202 is first converted into a constraint data structure by the generative language model. The constraint data structure is then parsed to extract the constraint parameters, and the generative language model is used to generate constraint checking code that checks whether possible solutions satisfy the individual constraint parameters. For example, each constraint parameter can be checked by a Python function, such as checking whether a candidate meeting time satisfies the user's given preference (i.e., "do not schedule before 2 PM"). Each constraint data structure also has an associated priority parameter, which can represent the priority constraint parameters output by the generative language model. Higher priorities correspond to more important constraints. Instead of requiring the user to specify precise numerical priority values ​​for each preference, the interactive constraint-satisfying agent 210 preserves an underlying list of priorities for the constraints and then converts it into weights such that constraints with higher priorities are satisfied before constraints with lower priorities.

[0039] The constraint-checking code generated by the generative language model 100 can be invoked by the constraint solver 220. The constraint solver can generate new candidate suggestions that satisfy some or all of the constraint parameters on the current constraint list. This hybrid framework leverages the benefits of both generative language models and constraint-satisfying algorithms, using the generative language model to flexibly embed natural language scheduling constraints into Python functions, and using the constraint solver to generate new time suggestions in response to the current constraints. This occurs in an interactive chat environment that allows users to naturally express their preferences without having to express them as formal constraints (e.g., code or logical expressions about variables) suitable for direct use by the constraint solver.

[0040] When users wish to schedule a meeting, they can first input information in natural language (such as a list of attendees, meeting duration, and / or time range) into a form. Once the user has entered the information, constraint solver 220 generates an initial time suggestion that maximizes the feasibility of the desired attendees. Interactive constraint satisfying agent 210 returns the initial time suggestion and initiates a chat, allowing the user to interact with the system to refine the suggested meeting time.

[0041] For each new chat message entered in the chat, the constraint manager 211 translates the request into an interactive constraint to satisfy the action that the agent 210 must take. For example, the constraint manager can use a request to select one of the available actions to prompt the generative language model 100. One such action is to add a new constraint, which involves translating natural language preferences into constraint-checking code, as described above. These functions are then used within the constraint solver 220 to generate new time suggestions until the user accepts the suggested time, at which point the meeting can be scheduled for the accepted time and meeting invitations can be created in the user's schedule and the schedules of other participants.

[0042] Constraint Manager When a user enters a new chat message, the constraint manager 211 can select the action to take, such as: Add Constraint: Generates a new schedule constraint and invokes constraint generator 217.

[0043] Change Priority: Changes the priority of existing constraints in the current constraint list.

[0044] Delete Constraint: Removes the given schedule constraint from the current constraint list.

[0045] Send a message to the user: Send a message back to the user.

[0046] Generate Suggestion: Calls constraint solver 220 to generate a new time suggestion and returns it to the user.

[0047] In some cases, the entire chat history between the user and the interactive constraint-satisfying agent 210, along with the current list of scheduling constraints, is provided as input to the constraint manager. This allows the action selected by the generative language model 100 to be based not only on the latest chat message but also on the context in which the message is placed. For example, if a user attempts a scheduling constraint (e.g., “How about a meeting on Thursday?”) and then decides to abandon the proposal (e.g., “Oh well”), the system can remove the corresponding scheduling constraint without any further follow-up questions.

[0048] Figures 3A to 3D An example constraint management prompt template 300 is shown, which can be adopted by constraint manager 211 to generate constraint management prompts for generative language model 100. Constraint management prompt template 300 includes an instruction section 302, an available action section 304, an action example section 306, a response format section 308, and a current information section 310.

[0049] The instruction section 302 conveys a request to the generative language model 100 for selecting an action to take. The available actions section 304 conveys a list of available actions that the generative language model can select from when responding. The action example section 306 conveys examples of user input, existing constraints ("preference list"), and actions to be taken in response. The response format section 308 specifies the format used by the generative language model 100 to respond. The current information section 310 includes information about the current scheduled conversation, such as the current chat history with the user, the current list of constraints, etc. For example, suppose a user types the text "I have to have a meeting before 11 a.m." into the current chat history. I have to meet before 11am This text can be added to a constraint management cue template to obtain constraint management cue words. Generative language models can respond with actions such as: {"Action": "Add", "Input": {"Constraint": "Meeting before 11:00 AM", "Priority": 1}} ({"ACTION": "ADD," "INPUT": {constraint": "Meeting before 11am", "priority": 1}}） Note that the "Add" action specifies two constraint parameters—the text string "Meet before 11 a.m." and a priority of "1". Because the generative language model uses a specified data format to express constraints, the constraint manager 211 can reliably extract constraint parameters from the output of the generative language model. Here, priority 1 means a mandatory constraint; for example, the generative language model has inferred that "have to" means the user cannot hold the meeting at another time.

[0050] As the user continues to interact with the interactive constraint-satisfying agent 210, the current chat history can be appended to the constraint management prompt template 300 to prompt the generative language model 100. Constraints can be added, removed, or modified based on the determination of the generative language model. The generative language model can also message the user with additional information; for example, if the user says "I can't have the meeting before 6 o'clock," the generative language model can send the user "6 am or 6 pm?" to further clarify before adding a new constraint. The generative language model can also decide to generate a suggestion that invokes the constraint solver, as described below.

[0051] constraint generator When constraint manager 211 decides to add a new scheduling constraint, data inspector 218 first checks whether the given scheduling constraint can be handled by interactive constraint-satisfying agent 210. Considering the diversity of user scheduling preferences and the breadth of external data required to integrate these preferences, the interactive constraint-satisfying agent may not handle all scheduling constraints. This check serves as a safeguard to determine whether the system has sufficient data to handle the constraint. If not, generative language model 100 can identify the missing data (which can be queried from the user) or explain why the given constraint cannot be handled. If the constraint can be handled, code generator 219 can use generative language model 100 to convert the constraint into constraint-checking code, which can be used to check whether candidate meeting times satisfy the constraint.

[0052] Figure 4A and Figure 4BAn example data check prompt template 400 is shown that can be used by data checker 218 to prompt generative language model 100. Data check prompt template 400 includes an instruction section 402, an example section 404, and a current information section 406. Instruction section 402 conveys instructions for generative language model 100 to determine whether an available data source provides the data needed to generate constraint check code for a given constraint. For example, the data checker may include code that populates the instruction section based on the availability of the information source (e.g., a schedule plugin). Example section 404 conveys an example of data checking that illustrates how the availability or unavailability of a data source determines whether constraint check code can be generated. Current information section 406 can be populated with a query related to the current constraint expressed by the user. For example, given the current user input "Billy can only hold meetings on days his wife doesn't work," generative language model 100 might output the following: Output: {{"response": "No", " rationale "We are unable to access Billy's wife's work schedule." Output: {{"response": "no", "rationale": "We do not have access to Billy's wife's work schedule."}} ) Code generator 219 can be invoked when the generative language model responds "yes" or otherwise indicates that the constraint parameters can be examined given an available data source. Figure 5A and Figure 5B A sample code generation prompt template 500 is shown that can be used by constraint manager 211 to prompt the generative language model 100. The code generation prompt template 500 includes an instruction section 502 instructing the generative language model to generate constraint checking code. An example section 504 includes examples of constraints and examples of corresponding constraint checking code to check whether the example constraints are satisfied. A current information section 506 can be populated based on the current constraints used to query the generative language model.

[0053] Note that the example in code generation prompt template 500 specifies a particular format for the generated constraint-checking code. As discussed in more detail below, the use of a consistent format allows the constraint-checking code to be invoked by constraint solver 220. For example, a function generated by a generative language model can conform to a specified function signature that defines the input and output parameters for the constraint-checking function generated by generative language model 100.

[0054] Constraint Solver Given a list of scheduling constraints and their associated priorities, constraint solver 220 attempts to find the candidate meeting time with the highest score, which is defined as a weighted sum of the constraints satisfied, where the weights are a function of the priorities determined by a generative language model, and whether a constraint is satisfied is determined by running constraint checking code generated by the generative language model.

[0055] The relevant constrained programming problem is formally defined below, and it is shown that the problem can be solved efficiently. Let... It is a set of candidate times or meetings. For example, for a one-hour meeting on Tuesday, list all one-hour blocks starting at one hour or half an hour between 8:00 AM and 6:00 PM (i.e., 8-9am, 8:30-9:30am, 9-10am 5-6pm ).make It is to select the time Schedule constraints are mapped to Boolean values ​​representing whether a time condition is met. As mentioned earlier, these constraints can be evaluated using code generated by a generative language model, such as Python functions. A set of candidate times exists as input to the constraint solver. Each has its own associated weight (or priority). of Schedule constraints Formally, the goal of a constraint solver is to solve optimization problems:

[0056] For actual meeting scheduling examples, candidate time sets The size is small enough that a brute-force approach can be used to score all candidate times and return the highest-scoring candidate time. For reference, a meeting scheduling instance with 100,000 candidate times and 10,000 scheduling constraints can be solved in 10 seconds on a single thread.

[0057] The constraint solver can also be configured to return different time suggestions. The constraint solver can return a set of... Instead of returning only a single suggestion with the best weighted score, the set of suggestions returned is more diverse. Here, diversity is defined as returning a set of suggestions. The sum of pairwise distances between each time point in the time interval:

[0058] in This is a distance function between two suggested times. The distance between two candidate times is defined as the logarithm of their difference in minutes plus one. To construct different time suggestions, a greedy algorithm can be used. The greedy algorithm filters candidate times, selecting only those with the best score. It begins within a certain timeframe. Then, select... To find the minimum value, such that at least one The criteria are satisfied once. After filtering the time, the set is greedily constructed by first selecting the earliest high-scoring time and iteratively adding new times that maximize the diversity of the current set.

[0059] To help provide transparency regarding suggested meeting times, each meeting time can be returned to the user along with an explanation of what constraints were met, such as visualizations, a one-sentence summary of its advantages and disadvantages, etc. To generate this summary, a list of scheduling preferences met and not met for each suggested time can be used to prompt the generative language model 100, instructing it to generate a one-sentence summary, etc. For example, here is a sample explanation for a time that meets two user preferences but fails to include one attendee: the time is before 11:00 AM on Tuesday, but Anton cannot attend.

[0060] Figure 6A and Figure 6B An example solution explanation prompt template 600 is shown that can be used by constraint manager 211 to prompt generative language model 100. Solution explanation prompt template 600 includes instruction section 602, which instructs the generative language model to generate explanations of which constraints are satisfied / not satisfied for a given suggestion. Example section 604 provides examples of explanations that can be generated from the corresponding satisfied and unsatisfied preferences. Current information section 606 can be populated with satisfied and unsatisfied constraints for the solution for which explanations are being generated.

[0061] Example User Interface The specific implementation described above was developed in a system called "MeetMate". Figure 7A A schedule user interface 700 is shown, which displays the user's weekly schedule with shaded time periods 702 that correspond to a suggested meeting time, which will be discussed further below. Figure 7B The chat history 710 involving a user attempting to schedule a meeting is shown. Although shown on a separate form, the scheduling user interface 700 can be displayed simultaneously with the chat history 710 (e.g., next to the chat history 710). The chat history conveys a natural language dialogue with the interactive constraint-satisfying agent 210. Note that messages from the user are shown in bold in the chat history.

[0062] like Figure 7B As shown, a user can initiate interaction by first specifying a list of attendees 711 and the desired meeting duration 712. These can be interpreted as initial constraints by the interactive constraint-satisfying agent 210, which responds with an initial suggestion 713 with a brief explanation. If the suggested time is satisfactory, the user can schedule the meeting by clicking the "Schedule" button 714. If the suggested time is unsatisfactory, the user can freely interact with the system through a message box. Each time a user sends a message, the interactive constraint-satisfying agent processes the message as described above and responds with a suggested time and / or message, in which case the user accepts the suggestion or iterates through interaction with the system. For example, the user expresses constraint 715, which can be processed as described above by adding a new constraint and solving it to generate a new suggestion 716. This can continue until the user accepts the suggestion, at which point the meeting can be added to the user's schedule and the schedules of other attendees.

[0063] As described above, in some cases, the constraint solver 220 can be configured to return multiple suggestions at once. Multiple suggestions can be selected based on methods that promote diverse suggestions (e.g., time diversity criteria such as the diversity formula provided above). Figure 7C An exemplary list of diverse suggestions 720 is shown, with suggestions 713, 721, and 722 varying depending on the day of the week and time of day (e.g., morning vs. afternoon). This approach encourages exploration of the solution space, thereby efficiently uncovering user preferences that are not explicitly expressed and quickly identifying acceptable solutions for the user.

[0064] Experimental results The following describes the experiments performed to evaluate the implementation of MeetMate. An evaluation benchmark was constructed to quantitatively assess information checking and constraint solving. This benchmark combines a synthetic scheduling environment with real-world scheduling preferences collected from user research. The synthetic scheduling environment was used to sample meeting scheduling scenarios without using personally identifiable information or private scheduling data. To generate this scheduling environment, the study used GPT-4 to construct a synthetic organization of 32 employees distributed across 4 different teams. The generative language model was also used to generate a series of existing meetings among the employees to populate their schedules. The number of meetings per employee was matched to their position (e.g., managers have a higher meeting load) and calibrated based on the number of meetings self-reported by users in the diary research. Furthermore, a dataset containing 75 new meeting instances was generated, representing new meetings that needed to be scheduled.

[0065] To generate a dataset of scheduling preferences, the results of the user study were processed. First, scheduling preferences and constraints derived during the study were processed to remove any unintentionally shared personally identifiable information (PII). To improve the generalization ability of the acquired preferences, all specific time, day of the week, or meeting-related information (e.g., attendees, event names) was removed and replaced with placeholders. These placeholders were then populated in one of two ways: by sampling from a uniform distribution of the potential inputs (e.g., randomly generating a day from five weekdays); or by filling in features relevant to a specific meeting instance (e.g., filling in the name of the meeting organizer). For each original preference from the diary study, three new preferences were generated by populating new meeting schedule instances with different placeholder values.

[0066] The processed preferences in the study were categorized into two types: requests that the current system can handle under the constraints of the synthetic scheduling environment; and requests that the current system cannot handle under the constraints of the synthetic scheduling environment. For example, since the synthetic environment does not contain any facility information, any preferences related to meeting room availability were labeled "unprocessable." These categories form the basis of a binary classification dataset (i.e., the "Safeguard Dataset"), through which the information inspector component in the system is evaluated. A subset of the dataset that can be processed by the current system was used to form a dataset for the coder component, called the code generation dataset.

[0067] The dataset was used to benchmark the performance of the current generative language of the information inspector component of the MeetMate system. Two different generative language models were evaluated: GPT-3 (text-davinci-003)

[11] and GPT-4

[52] . Two different wording strategies were also evaluated. During the initial experiments, the generative language models struggled to accurately extract the correct scheduling preferences when faced with long sentences containing reasoning processes. To address this, a rephraser component driven by the generative language model was introduced. Given an initial scheduling preference, the model was asked to restate it as a concise and clear scheduling constraint. It should be noted that this step is not required in the full implementation of the MeetMate system, as the constraint manager 211 can directly restate chat messages as scheduling constraints. The classification accuracy of each model and wording strategy on the reference dataset is reported below, which is defined as the proportion of instances that the model correctly predicts as either the instances that the current system can or cannot handle.

[0068] Figure 8Table 800 shows the results summarizing the accuracy of the two different models and prompt word strategies. When using the restater, both models achieve approximately 80% accuracy for the information inspection task. The restater does bring a small performance improvement to both models, highlighting the importance of transforming complex user queries into clear and unambiguous requests before generating code.

[0069] To evaluate the code generator 219 of the MeetMate system, constraint-checked code generated by a generative language model was compared with true correct implementations of each function generated by human software developers. Correct implementations followed a similar processing strategy to the original dataset. Implementations were generated only for each preference processed and containing placeholders from the diary research, with placeholders populated based on the specific meeting context or sampled values. The code generation component was evaluated for the following three metrics: Compilation success rate: The percentage of functions generated by each component that can be successfully imported by the Python interpreter.

[0070] Correctness-Precision: In successfully running code, the proportion of candidate times marked as satisfying a given preference that actually satisfy that preference.

[0071] Correctness-Recall: In successfully running code, the proportion of all candidate times that should actually be marked as satisfying a given preference are correctly marked by the code.

[0072] To calculate the precision and recall of the generated code, two sets of candidate times are generated for each preference in the code generation dataset. The first set, called the general dataset, contains all meeting times that meet the correct duration within the next 50 days. The second set, called the example dataset, restricts the candidate times to the time range of synthetic meeting schedule examples (spanning 2 to 14 days depending on the specific example) and only includes time periods that meet the correct duration. These two sets of datasets are evaluated to capture code generation errors that may not be exposed within a shorter timeframe but will become apparent over a longer time span. Figure 8 The report presents the performance of the two models and different prompting strategies on the five metrics mentioned above. Both generative language models can generate successfully compileable code in over 95% of cases, and exhibit excellent performance in precision and recall (both exceeding 90%). This fully demonstrates the feasibility of using generative language models to transform user scheduling preferences into accurate software representations.

[0073] Example System The proposed implementation can be executed on various devices in various scenarios. Figure 9 An example system 900 in which the proposed implementation can be adopted is shown, as discussed in more detail below.

[0074] like Figure 9 As shown, system 900 includes client devices 910, servers 920, 930, and 940 connected via one or more networks 950. Note that client devices can be mobile devices such as smartphones or tablets, or fixed devices such as desktop computers or server equipment. Similarly, servers can be implemented using various types of computing devices. In some cases, Figure 9 Any of the devices shown, especially servers, can be implemented in data centers, server farms, etc.

[0075] Figure 9 Some components of the devices shown herein may be referred to herein by reference numerals in parentheses. For the purposes of the following description, parentheses (1) indicate the presence of a given component on client device 910, (2) indicate the presence of a given component on server 920, (3) indicate the presence on server 930, and (4) indicate the presence on server 940. Unless identifying a specific instance of a given component, this document will generally refer to the component without parentheses.

[0076] Typically, devices 910, 920, 930, and / or 940 may have corresponding processing resources 901 and storage resources 902, which will be discussed in more detail below. The devices may also have various modules that use the processing and storage resources to perform the techniques discussed herein. Storage resources may include both persistent storage resources (such as magnetic or solid-state drives) and volatile storage devices (such as one or more random access memory devices). In some cases, modules are provided as executable instructions stored on a persistent storage device, loaded into a random access memory device, and read from the random access memory by the processing resources for execution.

[0077] Client device 910 may include a local application 911, which can be used to interact with the interactive constraint-fulfilling agent 210 on server 920. Various components of the interactive constraint-fulfilling agent can prompt the generative language model 100 on server 930 based on user input received from client device 910. The interactive constraint-fulfilling agent can also respond to the user based on responses output by the generative language model. In other words, the interactive constraint-fulfilling agent can act as an intermediary between the user and the generative language model, which not only utilizes the generative language model to communicate with the user but also utilizes the generative language model to generate formal representations of constraints and constraint-checking codes.

[0078] Server 940 may include compiler 941, which compiles or interprets code generated by code generator 219. The code may be executed during runtime 942 to examine data sources 943 and / or 944 to see if constraints are satisfied. The output of the executed constraint-checking code may be sent to constraint solver 220 of interactive constraint-satisfying agent 210. The output may be a Boolean value indicating whether a given candidate solution evaluated by the constraint solver satisfies a given constraint.

[0079] Example Method Figure 10 An example method 1000 based on the proposed concept is shown. As discussed in more detail below, method 1000 can be implemented on many different types of devices, for example, via one or more cloud servers, via client devices such as laptops, tablets, or smartphones, or via a combination of one or more servers, client devices, etc. Method 1000 begins at box 1002, where natural language user input is received. For example, a user message conveying preferences could be received. Preferences can relate to any kind of task, such as scheduling a meeting, finding an airline flight, or buying a car. Method 1000 continues at box 1004, where constraint management prompts are generated. As previously described, constraint management prompts can be generated from a template, for example, by populating the template with current information such as natural language input received at box 1002, previous chat history with the user, current constraint list, etc. Method 1000 continues at box 1006, where constraint management prompts are input into the generative language model. As previously described, constraint management prompts can specify the actions available to the generative language model and the format of the constraint data structure, according to which the individual constraint parameters are represented. An example of text constraint descriptions and priorities as constraint parameters has been described above.

[0080] Method 1000 continues at box 1008, where a constraint data structure is received from the generative language model. For example, as described above, the constraint data structure can be output by the generative language model when it selects the "Add Constraint" action from the available actions.

[0081] Method 1000 continues at box 1010, where the constraint data structure is parsed to extract constraint parameters, such as text constraint descriptions and priorities.

[0082] Method 1000 continues at box 1012, where the constraint solver processes constraint parameters. For example, the constraint solver may check whether available solutions (e.g., time periods) satisfy certain constraint parameters (e.g., participant feasibility, room availability, etc.). In some cases, box 1012 involves inputting code generation prompts into a generative language model to generate code for performing the checks. As previously mentioned, the constraint solver may also weight constraints based on priority parameters.

[0083] Method 1000 continues at box 1014, where it outputs candidate solutions identified by the constraint solver. If the user accepts a given candidate solution at decision box 1016, the method moves to box 1018, where it updates the data source using the accepted solution (e.g., by adding the accepted meeting to one or more user schedules). Otherwise, the method returns to box 1002, where it receives further natural language input.

[0084] Alternative implementation methods The aforementioned techniques are conveyed in the context of a user scheduling a meeting with other users. However, user preferences can be represented in natural language for a wider range of applications. For example, consider a network administrator who wants to perform scheduled maintenance on cloud servers in a data center, where the scheduled maintenance requires one hour of downtime for each server. Here, the network administrator might want to convey some technical constraints in natural language format, such as not shutting down servers running high-priority jobs. Interactive constraints satisfy that the agent has access to the data source of currently running jobs in the data center and can select which servers to shut down for maintenance based on which jobs are scheduled on which servers. In some cases, individual servers can be automatically shut down when the generative language model selects the "shut down servers" action from the list of available actions.

[0085] As another example, in the constraint management prompt example above, the example constraint data structure includes text-described constraint parameters and priority constraint parameters. However, various other types of constraint parameters are reasonable. For example, instead of integer priority parameters, a boolean value could be specified indicating whether the constraint is a hard constraint (value 1) or a soft constraint (value 0). In other implementations, the textual representation of constraints can be expressed in a more formal way. For example, Figure 3A The textual description of the constraint is shown as [No meeting in the morning], but this can also be represented as key-value pairs, such as ["morning", 0].

[0086] Furthermore, in some cases, generative language models can be employed to gather information to determine whether a given constraint can be satisfied. Consider a constraint for choosing a restaurant, such as "Jen is vegetarian." In some cases, a generative language model can convey whether a given restaurant offers vegetarian options; for example, for a chain restaurant, if the generative language model has acquired sufficient training data, it can learn whether the chain offers vegetarian options. Here, even if the interactive constraint satisfaction agent 210 may not have access to the data source of the restaurant menu, the generative language model can supplement the available data so that the constraint can be satisfied. It should also be noted that in this case, the task does not necessarily involve arranging a meeting at a restaurant, but simply choosing a restaurant. More generally, the disclosed techniques can be used to identify solutions that satisfy complex user preferences for various types of problems.

[0087] In another implementation, constraints can be stored for use in later tasks. For example, considering the scenario above, there is a constraint indicating that Jen is a strict vegetarian. When the interactive constraint satisfies agent 210's initial learning of Jen's preferences in this regard, a restaurant can be selected immediately using the disclosed techniques. Subsequently, the satisfaction of the interactive constraint can automatically apply that constraint to future tasks involving restaurant selection. More generally, a stored list of constraints can be maintained for each user, and the stored list of constraints can be applied to each task involving a given user.

[0088] Technical effect As mentioned above, many users are not adept at writing code or data structures that reflect their preferences in a way that is suitable for constraint solvers. While generative language models are very good at understanding user intent and engaging in dialogue with users, they are not skilled or efficient at solving users' complex constraint satisfaction problems.

[0089] The disclosed implementation bridges the capability gap between the constraint solver and the generative language model by using a few-shot prompting approach. By prompting the generative language model with examples of constraint data structures containing constraint parameters in a specified format, the generative language model can be conditionalized to represent user preferences in a format suitable for subsequent processing. In other words, the constraint data structure output by the generative language model can be easily parsed to identify parameters (such as text descriptions) that can be converted into constraint checking code, as well as other parameters (such as priorities) that can be used to weight the constraint satisfaction algorithm. Furthermore, the generative language model can exert programmatic control over the logical flow of constraint satisfaction, as it can choose which actions to take at specific points in the dialogue with the user.

[0090] Furthermore, a small number of sample hints can also be used for code generation itself. Therefore, code generated by a generative language model can conform to the function signatures expected by the constraint solver. Consequently, the constraint solver can directly call the code without modifying itself. Thus, the constraint solver's ability to solve new constraints can be flexibly extended by calling the new constraint-checking code generated by the generative language model. Moreover, note that even assuming the generative language model is capable of solving complex constraints, doing so using a generative language model would be considerably less efficient than using a constraint solver. Executing a generative language model may involve using multiple high-performance processors (e.g., GPUs) and hundreds of GB of RAM, while a constraint solver can typically run on a single CPU and a few GB or less of RAM.

[0091] More generally, few-sample prompts for generative language models allow for context-based learning of how to generate constraint data structures and constraint-checking codes. This is far more efficient than, for example, pre-training a generative language model using a corpus of constraint data structures and / or constraint-checking codes. Instead, once the initial generative model is trained, the few-sample prompting technique disclosed in this paper allows the model to learn how to generate constraint data structures and / or constraint-checking codes from the context of the prompt words. This requires significantly less training data and associated computational and memory resources compared to pre-training or tuning a generative language model.

[0092] Device implementation method As mentioned above Figure 9 The system 900 includes several devices, including client device 910, server 920, server 930, and server 940. It is also noted that not all device implementations can be shown, and other device implementations should be obvious to those skilled in the art from the above and below description.

[0093] As used herein, the terms “device,” “computer,” “computing device,” “client device,” and / or “server device” can mean any type of device having a certain amount of hardware processing power and / or hardware storage / memory capacity. Processing power can be provided by one or more hardware processors (e.g., hardware processing units / cores) capable of executing computer-readable instructions to provide functionality. Computer-readable instructions and / or data can be stored on storage devices (such as storage devices / memory and / or data storage devices). The term “system” as used herein can refer to a single device, multiple devices, etc. Storage resources can be internal or external to their associated corresponding devices. Storage resources can include any one or more of volatile or non-volatile memory, hard disk drives, flash memory devices, and / or optical storage devices (e.g., CDs, DVDs, etc.). As used herein, the term "computer-readable medium" can include signals. Conversely, the term "computer-readable storage medium" does not include signals. Computer-readable storage media includes "computer-readable storage devices." Examples of computer-readable storage devices include volatile storage media (such as RAM) and non-volatile storage media (such as hard disk drives, optical discs, and flash memory, etc.).

[0094] In some cases, the device is configured with general-purpose hardware processors and storage resources. The processor and storage devices can be implemented as separate components or integrated together, as in computational RAM. In other cases, the device may include a system-on-a-chip (SOC) design. In an SOC design implementation, the functionality provided by the device can be integrated onto a single SOC or multiple coupled SOCs. One or more associated processors can be configured to coordinate with shared resources (such as memory, storage devices, etc.) and / or one or more dedicated resources (such as hardware blocks configured to perform certain specific functions). Therefore, the terms “processor,” “hardware processor,” or “hardware processing unit” as used herein can also refer to a central processing unit (CPU), graphics processing unit (GPU), controller, microcontroller, processor core, or other types of processing devices suitable for implementation in both conventional computing architectures and SOC designs. Processors and storage devices can be implemented as separate components or integrated together, as in computational RAM.

[0095] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc. In some configurations, any modules / code discussed herein may be implemented in software, hardware, and / or firmware. In any case, modules / code may be provided during the manufacture of the device or by an intermediary preparing to sell the device to the end user. In other cases, the end user may install these modules / code later, such as by downloading executable code and installing it on the corresponding device.

[0096] It's also important to note that devices can typically have input and / or output capabilities. For example, computing devices can have various input mechanisms, such as keyboards, mice, touchpads, voice recognition, gesture recognition (e.g., using depth cameras such as stereo vision or time-of-flight camera systems, infrared camera systems, RGB camera systems, or using accelerometers / gyroscopes, facial recognition, etc.). Devices can also have various output mechanisms, such as printers and displays.

[0097] It should also be noted that the devices described herein can function independently or collaboratively to implement the described techniques. For example, the methods and functions described herein can be executed on a single computing device and / or distributed across multiple computing devices communicating via multiple networks 950. Without limitation, network 950 may include one or more local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0098] The above describes various examples. Additional examples are described below. One example includes a computer-implemented method comprising: receiving natural language input from a user, the natural language input specifying the user's preferences in natural language; generating constraint management prompts for a generative language model, the constraint management prompts being based on the natural language input and including instructions that request the generative language model to generate a constraint data structure representing the preferences according to a specified constraint data format; inputting the constraint management prompts into the generative language model; receiving from the generative language model the constraint data structure generated by the generative language model, the constraint data structure being in the specified constraint data format; parsing the constraint data structure generated by the generative language model to extract constraint parameters; processing the constraint parameters using a constraint solver to identify candidate solutions that satisfy at least some of the constraint parameters; outputting candidate solutions to a user; and updating a specific data source using the accepted solution in response to user input identifying an accepted solution from the candidate solutions.

[0099] Another example may include any of the above and / or the following examples, wherein the method further includes: identifying available data sources for the generative language model; and providing the generative language model with data inspection prompts that instruct the generative language model to determine whether constraint parameters can be inspected given the available data sources.

[0100] Another example may include any of the above and / or the following examples, wherein the method further includes: in response to a response indication from the generative language model that the constraint parameters can be checked given an available data source, providing a code generation prompt to the generative language model, the code generation prompt instructing the generative language model to generate constraint checking code to check the constraint parameters; receiving the constraint checking code from the generative language model; and executing the constraint checking code generated by the generative language model to determine whether possible solutions satisfy the constraint parameters.

[0101] Another example may include any of the above and / or the following examples, where the execution includes: the constraint solver calling the constraint check code.

[0102] Another example may include any of the above and / or the following examples, wherein the method further includes one or more examples of including constraint checking code in the code generation prompt.

[0103] Another example may include any of the above and / or the following examples, wherein the constraint solver is configured to execute a constraint checking function with a specified format, and one or more examples of the constraint checking code are in the specified format.

[0104] Another example may include any of the above and / or the following examples, wherein the constraint parameters include constraint priorities, and processing the constraint parameters using a constraint solver includes weighting the corresponding constraints based on their priorities.

[0105] Another example could include any of the examples above and / or below, where constraint management prompts specify a list of available constraint management actions for a generative language model to select from, based on natural language input received from the user.

[0106] Another example could include any of the examples above and / or below, where the list of available constraint management actions includes adding a new constraint, changing the priority of an existing constraint, deleting a constraint, sending a message to the user, and generating a new candidate solution.

[0107] Another example may include any of the above and / or the following examples, where the constraint management prompt includes an example of available constraint management actions and a specified constraint data format.

[0108] Another example may include any of the above and / or the following examples, wherein the method further includes: generating constraint management prompts from a template of an example having available constraint management actions.

[0109] Another example may include any of the above and / or the following examples, wherein the method further includes: including the user's conversation history in the constraint management prompt.

[0110] Another example may include any of the above and / or the following examples, wherein the method further includes including a list of previously generated constraints in the constraint management prompt.

[0111] Another example may include a system comprising: a hardware processing unit; and a storage resource storing computer-readable instructions that, when executed by the hardware processing unit, cause the system to: receive natural language input from a user, the natural language input specifying the user's preferences in natural language; generate constraint management prompts for a generative language model, the constraint management prompts being based on the natural language input and including instructions that request the generative language model to generate a constraint data structure with constraint parameters representing the user's preferences; generate code generation prompts for the generative language model, the code generation prompts instructing the generative language model to generate constraint checking code that checks whether possible solutions satisfy the constraint parameters; execute the constraint checking code using a constraint solver to identify candidate solutions that satisfy at least some of the constraint parameters; output candidate solutions to the user; and update a specific data source using the accepted solution in response to user input identifying an accepted solution from the candidate solutions.

[0112] Another example could include any of the examples above and / or below, where the preference involves scheduling a meeting for the user and the specific data source is a schedule associated with the user.

[0113] Another example could include any of the examples above and / or below, where the constraint solver evaluates constraint parameters on the user's schedule and other users' schedules to identify candidate solutions.

[0114] Another example may include any of the above and / or the following examples, wherein computer-readable instructions, when executed by a hardware processing unit, enable the system to: promote diversity of candidate solutions by evaluating possible solutions according to one or more time diversity criteria.

[0115] Another example may include any of the above and / or the following examples, wherein computer-readable instructions, when executed by a hardware processing unit, cause the system to: input a solution explanation prompt word to a generative language model, the solution explanation prompt word instructing the generative language model to generate a solution explanation for a particular candidate solution; receive a solution explanation from the generative language model, the solution explanation indicating at least some preferences in the preferences that are satisfied by the particular candidate solution; and output a solution explanation to a user.

[0116] Another example may include a computer-readable storage medium storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform actions including: receiving natural language input from a user, the natural language input specifying the user's preferences in natural language; generating a constraint management cue word for a generative language model, the constraint management cue word including preferences and instructions, the instructions requesting the generative language model to generate a constraint data structure representing preferences according to a specified constraint data format; inputting the constraint management cue word into the generative language model; receiving from the generative language model the constraint data structure generated by the generative language model, the constraint data structure being in the specified constraint data format; parsing the constraint data structure generated by the generative language model to extract constraint parameters; processing the constraint parameters using a constraint solver to identify candidate solutions that satisfy at least some of the constraint parameters; outputting candidate solutions to the user; and updating a specific data source using the accepted solution in response to user input identifying an accepted solution from the candidate solutions.

[0117] Another example may include any of the above and / or the following examples, wherein the computer-readable storage medium further includes: generating code generation prompts for a generative language model, the code generation prompts instructing the generative language model to generate constraint checking code, the constraint checking code checking whether possible solutions satisfy constraint parameters; and using a constraint solver to execute the constraint checking code to identify candidate solutions.

[0118] in conclusion Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms of implementing the claims, and other features and actions that a person skilled in the art will recognize are intended to fall within the scope of the claims.

Claims

1. A computer-implemented method (1000), comprising: (1002) Natural language input is received from the user, the natural language input specifying the user's preferences in natural language; Generate (1004) a constraint management prompt for a generative language model, the constraint management prompt being based on the natural language input and including an instruction that requests the generative language model to generate a constraint data structure representing the preference according to a specified constraint data format; Input the constraint management prompt word (1006) into the generative language model; Receive (1008) the constraint data structure generated by the generative language model, wherein the constraint data structure adopts the specified constraint data format; Parse (1010) the constraint data structure generated by the generative language model to extract constraint parameters; The constraint parameters (1012) are processed using a constraint solver to identify candidate solutions that satisfy at least some of the constraint parameters; Output the candidate solution (1014) to the user; and In response to user input identifying the accepted solution from the candidate solutions, the specific data source is updated (1018) using the accepted solution.

2. The method according to claim 1, further comprising: Identify the available data sources for the generative language model; as well as The generative language model is provided with data inspection prompts that instruct the generative language model to determine whether the constraint parameters can be inspected given the available data source.

3. The method according to claim 2, further comprising: In response to a response from the generative language model indicating that the constraint parameter can be checked given the available data source, a code generation prompt is provided to the generative language model, the code generation prompt instructing the generative language model to generate constraint checking code that checks the constraint parameter; Receive the constraint checking code from the generative language model; as well as The constraint checking code generated by the generative language model is executed to determine whether possible solutions satisfy the constraint parameters.

4. The method of claim 3, wherein the execution comprises: The constraint check code is called by the constraint solver.

5. The method according to claim 4, further comprising: The code generation prompt includes one or more examples of constraint check code.

6. The method of claim 5, wherein the constraint solver is configured to execute a constraint checking function having a specified format, and the one or more examples of the constraint checking code adopt the specified format.

7. The method of claim 6, wherein the constraint parameters include constraint priorities, and processing the constraint parameters using the constraint solver comprises: The corresponding constraints are weighted based on the constraint priority.

8. The method of claim 1, wherein the constraint management prompts specify a list of available constraint management actions for the generative language model to select from based on the natural language input received from the user.

9. The method of claim 8, wherein the list of available constraint management actions includes adding a new constraint, changing the priority of an existing constraint, deleting a constraint, sending a message to the user, and generating a new candidate solution.

10. The method of claim 9, wherein the constraint management prompt includes an example of the available constraint management actions and the specified constraint data format.

11. The method of claim 10, further comprising: The constraint management prompt is generated from the template of the example that has the available constraint management actions.

12. The method of claim 11, further comprising: The constraint management prompt includes the user's conversation history.

13. The method of claim 12, further comprising: The constraint management prompt includes a list of previously generated constraints.

14. A system (900, 920) comprising: Hardware processing unit (901); as well as Storage resource (902) stores computer-readable instructions that, when executed by the hardware processing unit, cause the system to: Receive natural language input from a user, wherein the natural language input specifies the user's preferences in natural language; Generate constraint management prompts for a generative language model, the constraint management prompts being based on the natural language input and including instructions that request the generative language model to generate a constraint data structure with constraint parameters representing the user's preferences; Generate code generation prompts for the generative language model, the code generation prompts instructing the generative language model to generate constraint checking code, the constraint checking code checking whether possible solutions satisfy the constraint parameters; The constraint checking code is executed using a constraint solver to identify candidate solutions that satisfy at least some of the constraint parameters. Output the candidate solution to the user; and In response to user input identifying the accepted solution from the candidate solutions, the specific data source is updated using the accepted solution.

15. The system of claim 14, wherein the preference relates to scheduling a meeting for the user, and the specific data source is a schedule associated with the user.

16. The system of claim 15, wherein the constraint solver evaluates the constraint parameters on the user's schedule and other users' schedules to identify the candidate solutions.

17. The system of claim 16, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: The diversity of the candidate solutions is promoted by evaluating the possible solutions according to one or more time diversity criteria.

18. The system of claim 14, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: The solution explanation prompts are input into the generative language model, which instructs the generative language model to generate a solution explanation for a specific candidate solution; The solution explanation is received from the generative language model, the solution explanation indicating at least some preferences among the preferences that are satisfied by the particular candidate solution; as well as The solution explanation is then output to the user.

19. A computer-readable storage medium (902) storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform an action, the action including: Receive natural language input from a user, wherein the natural language input specifies the user's preferences in natural language; Generate constraint management prompts for a generative language model, the constraint management prompts including the preference and the instruction, the instruction requesting the generative language model to generate a constraint data structure representing the preference according to a specified constraint data format; The constraint management prompts are input into the generative language model; Receive the constraint data structure generated by the generative language model from the generative language model, wherein the constraint data structure adopts the specified constraint data format; Parse the constraint data structure generated by the generative language model to extract constraint parameters; The constraint parameters are processed using a constraint solver to identify candidate solutions that satisfy at least some of the constraint parameters. Output the candidate solution to the user; and In response to user input identifying the accepted solution from the candidate solutions, the specific data source is updated using the accepted solution.

20. The computer-readable storage medium of claim 19, further comprising: Generate code generation prompts for the generative language model, the code generation prompts instructing the generative language model to generate constraint checking code, the constraint checking code checking whether possible solutions satisfy the constraint parameters; as well as The constraint check code is executed using the constraint solver to identify the candidate solutions.

Images