Evolving prompts for neural networks

Abstract

Systems, methods, and computer program code for generating a task-prompt for inclusion in a model input for controlling a model to perform a task. An evolutionary process is used to evolve prompts that increasingly improve the extraction of knowledge from a machine learned model. Some machine learned model, such as Large Language Models, can consume significant computing resources, and implementations of the described techniques are configured to use parallel processing in a way that facilitates makes efficient use of these resources.

Description

EVOLVING PROMPTS FOR NEURAL NETWORKSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No. 63 / 586,323, filed on September 28, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND

[0002] This specification relates to processing data using machine learning models.

[0003] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0004] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.SUMMARY

[0005] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that enable a user to evolve task-prompts for input to a (trained) machine learning model, such as a Large Language Model (LLM). Some models, such as LLMs, already encode knowledge about a vast array of problems. However, it is not always straightforward to use or extract that knowledge. The techniques described herein provide a mechanism to evolve prompts that increasingly improve the extraction of knowledge from a machine learned model. Further, processing using Large Language Models can consume significant computing resources and some implementations of the described techniques are configured to facilitate parallel processing in a way that makes efficient use of these resources.

[0006] In one example, a computer-implemented method for generating a task-prompt for inclusion in a model input for controlling a model to perform a task comprises performing an evolutionary process using first and second processes. The evolutionary process comprises a plurality of evolutionary rounds performed on a population. The model may be a neuralnetwork. The model may be a trained model (e.g. a trained neural network) trained to perform the task.

[0007] The first process comprises providing a task-prompt mutation input as an input to a task-prompt mutation model to generate a new task-prompt. The task-prompt mutation input comprises a mutation-prompt comprising an instruction to generate a new task-prompt based on at least one individual of the population. Generating a new task-prompt based on an individual of the population may mean generating a new task-prompt based on any property of, or associated with, the at least one individual. The at least one individual of the population may be an individual of the current population or of a previous population. The task-prompt mutation model may be a neural network, referred to as a task-prompt mutation neural network. The task-prompt mutation model may be a trained model, e.g. a trained task-prompt mutation neural network. The new task-prompt may be referred to as an evolved task-prompt.

[0008] The second process comprises providing a mutation-prompt mutation input as input to a mutation-prompt mutation model to generate a new mutation-prompt. The mutation-prompt mutation input comprises a mutation-prompt and a hypermutation-prompt comprising an instruction to modify the mutation-prompt. The mutation-prompt may be referred to as a mutation operator herein. The mutation-prompt mutation model may be a neural network, referred to as a mutation-prompt mutation neural network. The mutation-prompt mutation model may be a trained model, e.g. a trained mutation-prompt mutation neural network. The new mutation-prompt may be referred to as an evolved mutation-prompt.

[0009] Performing the evolutionary process comprises performing, for at least some of the evolutionary rounds, the first process to generate a new task-prompt to replace a task-prompt in the population. Performing the evolutionary process also comprises performing, for at least some of the evolutionary rounds, the second process to generate a new mutation-prompt for use in performing the first process. Performing the evolutionary process further comprises evaluating a fitness of task-prompts of at least some individuals of the population and selecting a task-prompt from the population based on respective finesses of the task-prompts.

[0010] In another example, a computer-implemented method for generating a task-prompt for inclusion in a model input for controlling a model to perform a task comprises receiving a task specification and determining, based on the task specification, an initial task-prompt. The model may be a neural network. The model may be a trained model (e.g. a trained neural network) trained to perform the task.

[0011] In this example, the computer-implemented method further comprises providing a mutation-prompt mutation input as input to a mutation-prompt mutation model to generate a mutation-prompt. The mutation-prompt mutation input comprises an initial mutation-prompt and a hypermutation-prompt comprising an instruction to modify the mutation-prompt. The mutation-prompt may be referred to as a mutation operator herein. The mutation-prompt mutation model may be a neural network, referred to as a mutation-prompt mutation neural network. The mutation-prompt mutation model may be a trained model, e.g. a trained mutationprompt mutation neural network.

[0012] In this example, the computer-implemented method further comprises providing the mutation-prompt and the initial task-prompt to a task-prompt mutation model in a task-prompt mutation input to generate a new task-prompt. The task-prompt mutation model may be a neural network, referred to as a task-prompt mutation neural network. The task-prompt mutation model may be a trained model, e.g. a trained task-prompt mutation neural network.

[0013] In another example, a computer-implemented method for performing a task comprises receiving a task specification, determining, based on the task specification, an initial taskprompt, receiving an initial mutation-prompt, generating a mutation-prompt comprising mutating the initial mutation-prompt based on a hypermutation-prompt, wherein the hypermutation-prompt comprises an instruction to modify the mutation-prompt, and generating a new task-prompt comprising mutating the initial task-prompt based on the mutation-prompt.

[0014] The methods described herein can be implemented on a system of one or more computers. The system may also comprise one or more storage devices storing instructions that, when executed, can cause one or more computers to perform the operations of any of the methods. The storage devices may be communicatively coupled to the one or more computers. The storage devices may be implemented separately, e.g. as one or more computer storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of any of the methods.

[0015] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0016] Mutation of the mutation-prompt (e.g. using the second process) has been found to be particularly effective in evolving new, more effective task-prompts. By evolving mutationprompts, the system is self-referential in that it not only generates improved task-prompts but also improves the way that it generates improved task-prompts.

[0017] Where the evolutionary process is performed for a particular domain, such as in one or more of the domains described above, the system finds prompt strategies that improve a task model’s ability to derive solutions to problems in that domain. The system is also general purpose in that it is able to evolve task-prompts for many different domains.

[0018] In some implementations, the first and second processes may advantageously be performed in parallel. In some implementations both the first process and the second process may be performed for more than one member of the population in parallel. In some implementations, evaluating the fitness of task-prompts of at least some individuals of the population can be performed in parallel with one or more of the first and second processes. The present techniques have been devised to exploit the ability of modern computer systems to perform highly parallel processes, to take advantage of parallel computing hardware, and the benefits that using such hardware can bring.

[0019] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 shows an example of a prompt evolution system;

[0021] Figure 2 is a flow diagram of an example method for generating a task-prompt;

[0022] Figure 3 depicts an initialization process for populating a population;

[0023] Figure 4 is a flow diagram of another example method for generating a task-prompt;

[0024] Figure 5 is a flow diagram of a method for performing a task;

[0025] Figure 6 depicts the fitness of a population of task-prompts over a number of evolutionary rounds.

[0026] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0027] Prompting a model, for example a large language model (LLM), is central to its downstream performance. Described herein is a prompt evolution system and associated methods, which provide a mechanism for evolving and adapting prompts for a given domain. A population of task-prompts (e.g. prompts for a task in a certain domain) can be initialized, mutated, and evaluated for fitness over multiple generations so as to evolve the task-prompts. According to methods and systems described herein, the mutation of the task-prompts isgoverned by mutation-prompts which the system also generates, mutates and evolves in a self- referential way. In this way, the mechanism allows for the improvement of task-prompts and also the improvement of the mutation-prompts that improve the task-prompts.

[0028] A model trained to perform a task in a particular domain may be trained using training data including a batch of question (Q) and answer (A) pairs. According to the examples described herein, during inference the model can be provided with a task-prompt P, which is a string used to condition the context in advance of some further input question Q. The taskprompt P is intended to ensure that a better (e.g. more correct) response A is provided than if the question Q had been presented in the absence of the task-prompt P. The prompt evolution system and associated methods herein enable the generation and mutation of task-prompts P according to an evolutionary algorithm, by conditioning the evolution on mutation-prompts M. The prompt evolution system and associated methods herein also enable the generation and mutation of the mutation-prompts M. By evolving mutation-prompts M, the system is self- referential in that it not only generates improved task-prompts, but also improves the way that it generates improves task-prompts.

[0029] Figure 1 shows an example of a prompt evolution system 100. The prompt evolution system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0030] The prompt evolution system 100 comprises a task-prompt mutation model 110. The task-prompt mutation model 110 is configured to receive a task-prompt mutation input 112 and generate a new task-prompt 114. The process performed by the task-prompt mutation model 110 may be referred to as a first process 111. The task-prompt mutation input 112 can include a task-prompt P and / or a mutation-prompt M, although the origins of these task-prompts P and mutation-prompts M can vary depending on the example implementation, as described in more detail below.

[0031] In some implementations, the task-prompt mutation model 110 is a LLM. Using the observation that LLMs can be used to generate variations of input text, the task-prompt mutation input 112 includes an (initial) task-prompt P and a mutation-prompt M. The taskprompt mutation model 110 performs an evolutionary algorithm by generating a mutated taskprompt P’ based on the initial task-prompt P and conditioned on the mutation-prompt M. The task-prompt mutation model 110 outputs the mutated task-prompt P’ as the new task-prompt

[0032] The prompt evolution system 100 also comprises a mutation-prompt mutation model 120. The mutation-prompt mutation model 120 is configured to receive a mutation-prompt mutation input 122 and generate a new mutation-prompt 124. The new mutation-prompt 124 may be referred to as an evolved mutation-prompt 124. The process performed by the mutationprompt mutation model 120 may be referred to as a second process 121. The generation of the new mutation-prompt 124 using the second process 121 can be referred to as hypermutation.

[0033] The prompt evolution system 100 also includes a population 130. The population 130 contains multiple entries, each referred to as an individual. The population 130 may be stored on any suitable storage system associated with the prompt evolution system 100, for example distributed across one or more memories associated with the one or more computers on which the prompt evolution system 100 is implemented. As will be described in more detail below, depending on the implementation each individual may comprise one or more different data types. In this example, each of the individuals of the population 130 includes a task-prompt 131 and a mutation-prompt 132. In an example where there are n individuals within the population 130, each individual may have a respective task-prompt 131-1... 131-n and each individual may have a respective mutation-prompt 132-1 ... 132-n. Each individual may include multiple task-prompts, for example two or more task-prompts. In some implementations, as described in more detail below, one or more individuals may comprise one or more task-prompts and no mutation-prompt or one or more mutation-prompts and no taskprompt, which may be particularly appropriate for initialization processes where prompts are being generated for the first time, also known as a zero-order prompt generation process. Furthermore, one or more individuals of the population 130 may include other features, for example a context. The context may include one or more reasoning paths (i.e. Chain of Thought reasoning paths). The reasoning paths may be reasoning paths that were used to generate the task-prompt of the individual.

[0034] The task-prompts 131 of individuals in the population 130 can have an associated fitness. Methods of evaluating the fitness of a particular task-prompt 131 are discussed below.

[0035] The prompt-evolution system 100 can be operated to generate a task-prompt for inclusion in a model input for controlling a model 140 to perform a task. For example, the model 140 may be configured to receive a query Q and a task-prompt 131 and output an answer A. In some examples, the model 140 may be configured to receive multiple queries Q and multiple task-prompts 140. For example, the model may receive a first query and a first taskprompt (e.g. associated with a particular individual) and generate a first intermediate output.The model 140 may then receive the first query, the first task-prompt, the first intermediate output, a second task-prompt (e.g. associated with the same particular individual) and generate a second intermediate output conditioned on the first query, the first task-prompt, the first intermediate output and the second task-prompt. The model 140 may also receive a third query, the first task-prompt, the first intermediate output, the second task-prompt and the second intermediate output and output an answer A. The model 140 can comprise a neural network. The model 140 can be, for example, a LLM. The model 140 may be referred to as a task model. While the model 140 is depicted in Figure 1, it should be understood that the model 140 is not part of the prompt-evolution system 100. Rather, an output of the prompt-evolution system 100 (e.g. a task-prompt 131) can be provided as an input to the model 140, which may be external to the prompt-evolution system. The prompt-evolution system 100 can be operated for a plurality of evolution rounds.

[0036] In use, the task-prompt mutation model 110 performs a first process 111. In the first process 111, the task-prompt mutation model 110 receives a task-prompt mutation input 112. The task-prompt mutation input 112, in this example, comprises an initial task-prompt 312 and a mutation-prompt 132. The mutation-prompt 132 comprises an instruction to generate a new task-prompt 114 based on at least one individual of the population 130. Generating a new taskprompt 114 based on an individual of the population may mean generating a new task-prompt 114 based on any property of, or associated with, the individual of the population 130. For example, the property may be a task-prompt 131 associated with an individual of the population 130. As such, the task-prompt 312 included in the task-prompt mutation input 112 may be a task-prompt 131 from the population 130.

[0037] The mutation-prompt may contain an instruction to generate a new task-prompt based on the initial task-prompt. Examples of mutation-prompts include: “imagine you need to follow this instruction. What would you tell yourself if you wanted to be the best in the world at it”, “Request more detailed responses: if the original prompt is ‘describe X’, the improved version could be, ‘Describe X, focusing on its physical features, historical significance and cultural relevance” and “If you have two related prompts, you can combine them to create a more complex and engaging question”.

[0038] A task specification may be provided by a user input or may be selected from a list based on a problem domain of the task. The initial-task-prompt may be the task specification itself, or may be a task-prompt selected or generated based on the task specification.

[0039] For at least some of the evolutionary rounds, performing the first process 111 may comprise selecting an individual from the population 130 and including in the task-prompt mutation input 112 the task-prompt 131 of the selected individual (e.g. task-prompt 131-a associated with individual a) to generate an new (evolved) task-prompt 114. In some implementations the task-prompt selected to be included in the task-prompt mutation input 112 may be the winner of a tournament among a plurality of individuals in the population 130. Such tournaments may be performed in parallel, for example a first tournament may be performed on first parallel hardware and a second tournament may be performed on second parallel hardware configured to operate in parallel to the first parallel hardware. In this way, the clock time of running such tournaments may be reduced by exploiting the parallel processing capabilities of a computer, leading to a reduced computational burden relating to selecting an individual. The tournament may be a binary tournament, although it will be appreciated that the tournament may be any other form of tournament as will be known to the skilled person. Selecting an individual from the population 130 may comprise comparing two or more individuals from the population and 130 selecting the individual comprising a task-prompt with the highest fitness of the selected individuals. A task-prompt may have a higher fitness than another task-prompt when it results in more correct answers in a set of training tasks than the other task-prompt.

[0040] The task-prompt 131 in the task-prompt mutation input 112 may be one of a plurality of task-prompts in the task-prompt mutation input 112. That is, the task-prompt mutation input 112 may comprise more than one task-prompt.

[0041] The mutation-prompt 132 included in the task-prompt mutation input 112 may also be associated with an individual of the population 130. For example, the mutation-prompt 132 may be one of the mutation-prompts 132-1 . . .n from the population 130. The mutation-prompt 132 may be the mutation-prompt 132 associated with the same individual as the individual associated with the selected task-prompt 131 from the population 130 (e.g. if task-prompt 131- a is selected, associated with individual a, a corresponding mutation-prompt 132-a also associated with the same individual a). Alternatively, or in addition, and as described in more detail below, the mutation-prompt 132 may be generated by a second process 121 performed by the mutation-prompt mutation model 120.

[0042] In at least some of the evolutionary rounds, once the new task-prompt 114 has been generated, at least one of the individuals of the population 130 can be replaced with a new individual comprising the new task-prompt 114. For example, the method may includereplacing an individual having a lower fitness than at least one other individual in the population 130. In other examples, the task-prompt 131 that is replaced may be a task-prompt 131 that is below a threshold fitness. In other examples, the task-prompt 131 that is replaced may be a task-prompt 131 that is above a threshold similarity to at least one other task-prompt 131 in the population.

[0043] In the second process 121, the mutation-prompt mutation input 122 comprises a mutation-prompt 132 and a hypermutation-prompt 133. The hypermutation-prompt 133 comprises an instruction to modify the mutation-prompt 132. As such, the mutation-prompt mutation model 120 receives the mutation-prompt 132 and an instruction to modify the mutation-prompt 132 and generates a new (evolved) mutation-prompt 124.

[0044] For at least some of the evolutionary rounds, the second process 121 may be performed to generate a new mutation-prompt 124 and the new mutation-prompt 124 may then be used in the first process 111. That is, the task-prompt mutation input 112 may comprise a task-prompt 131 (e.g. from the population 130) and a new mutation-prompt 124 generated by the mutationprompt mutation model 120. In this way, it is possible to evaluate the influence of the hypermutation via the new mutation-prompt 124 on the quality (e.g. fitness) of the new taskprompt 114 that is generated in the first process 111. The new mutation-prompt 124 may be used in the first process 111 shortly (e.g. in the same evolutionary round or in a next evolutionary round) after generation.

[0045] In some examples, performing the second process 121 may comprise selecting an individual from the population 130 and including in the mutation-prompt mutation input 122 the mutation-prompt 132 of the selected individual to generate an evolved (new) mutationprompt 124. Selecting an individual from the population 130 may comprise comparing two or more individuals from the population 130 and selecting the individual comprising a taskprompt 131 with the highest fitness of the selected individuals. The method may comprise replacing, in the population 130, the individual comprising a task-prompt 131 with a lower fitness with a new individual comprising the evolved mutation-prompt 124. In some examples, the new individual comprises both the evolved task-prompt 114 and the evolved mutationprompt 124.

[0046] The task-prompt mutation model 110 and the mutation-prompt mutation model 120 may each be the same model. The task-prompt mutation model 110 and the mutation-prompt mutation model 120 may be the same instance of the same model. That is, the task-prompt mutation input 112 and the mutation-prompt mutation input 122 may be provided to the samemodel. Alternatively, the task-prompt mutation model 110 and the mutation-prompt mutation model 120 may be different models. For example, the mutation-prompt mutation model 120 may be specifically used only to perform the second process 121 and not used to perform the first process 111, and vice versa. The task-prompt mutation model 110 and mutation-prompt mutation models 120 may operate on different computer systems. The task-prompt mutation model 110 and the mutation-prompt mutation model 120 may comprise one or more sequence models, such as a (trained) Large Language Model. Large Language Models are discussed in more detail below.

[0047] The present techniques have been devised to exploit the ability of modern computer systems to perform highly parallel processes. More particularly the process of generating a task-prompt is divided into two separate evolutionary processes that can be implemented by separate models (neural networks), i.e. the task-prompt model and the mutation-prompt model. These models can be operated in parallel on different, parallel computing hardware.

[0048] Thus, by configuring the system to use first and second processes on separate models that can be can run in parallel, the system is adapted to parallel computing hardware. Further, by configuring the process of generating a task-prompt in a hierarchical manner, with a taskprompt model and a hypermutation model, i.e. the mutation-prompt model that mutates the mutation-prompts, it is possible to search a space of possible prompts much more effectively and efficiently. That is, implementations of the method are specifically configured to take advantage of parallel computing hardware, and the benefits that using such hardware can bring. Such an arrangement also provides implementation flexibility. For example in some implementations the mutation-prompt model can be a larger model, e.g. a neural network with a greater number of learnable / trained parameters such as weights, than the task-prompt mutation model (neural network). This can be advantageous in that a more powerful model, e.g. LLM, can be used to drive the hypermutations.

[0049] Thus, for example, the method can be implemented in a parallel processing system comprising a plurality of sets of hardware devices. Each set of hardware devices comprises one or more hardware computing devices or processors, and the sets of hardware devices and / or processors are configured to operate in parallel. The method can be configured so that a first processor implements the task-prompt mutation model 110 and performs the first process 111, and a second processor implements the mutation-prompt mutation model 120 and performs the second process 121. In some implementations models comprise neural networks and themutation-prompt mutation model has more parameters, e.g. weights, than the task-prompt mutation model.

[0050] Additionally or alternatively, the fitness of task-prompts may be evaluated in parallel to other processes. For example, the fitness of task-prompts may be evaluated using a third processor configured to operate in parallel with one or more of the first or second processors performing the first and second processes 111, 121.

[0051] Additionally or alternatively, sub-processes within the process of evaluating the fitness of task-prompts may be performed in parallel. For example, the evaluation of the fitness of a first task-prompt may be performed in parallel to the evaluation of the fitness of a second taskprompt. In this example, the evaluation of the first and second task-prompts may be performed on first and second parallel hardware, respectively. The parallelization fitness evaluation of multiple task prompts is described in more detail further below.

[0052] Figure 2 is a flow diagram of an example method 200 for generating a task-prompt. For convenience, the method 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a prompt evolution system, e.g., the prompt evolution system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the method 200. The method 200 may be referred to as a prompt evolution method or process.

[0053] The method 200 comprises performing an evolutionary process using a first process 111 and a second process 121. The method 200 comprises a plurality of evolutionary rounds performed on a population, e.g. the population 130 described with reference to FIG. 1.

[0054] The first process 111 includes providing 213 a task-prompt mutation input as an input to a task-prompt mutation model to generate 214 a new task-prompt. As described above, the task-prompt mutation input comprises a mutation-prompt comprising an instruction to generate a new task-prompt based on at least one individual of the population. The property may be a task-prompt associated with the individual. Additionally or alternatively, the property may be a context (e.g., a reasoning path) associated with the individual as will be described in more detail below.

[0055] The second process 121 includes providing 223 a mutation-prompt mutation input as input to a mutation-prompt mutation model to generate 224 a new mutation-prompt. The mutation-prompt mutation input comprises a mutation-prompt and a hypermutation-prompt comprising an instruction to modify the mutation-prompt.

[0056] In some implementations, the first and second processes 111, 121 may advantageously be performed in parallel. In some implementations both the first process 111 and the second process 121 may be performed for more than one member of the population in parallel. The advantages of parallel processing, and how they apply to the systems and methods described herein, are described in more detail above.

[0057] In at least some of the evolutionary rounds, the method 200 includes, following performance of the first process 111, replacing 215 the task-prompt in the population (e.g. the task-prompt associated with the task-prompt mutation input used in the first process 111 in the present evolutionary round) with the new task-prompt generated using the first process 111. The step of replacing 215 may comprise replacing, in the population, at least one of the two or more individuals with a new individual comprising the new task-prompt. For example, the method may include replacing an individual having a lower fitness than the individual with the highest fitness.

[0058] In at least some of the evolutionary rounds, the method 200 includes, following performance of the second process 121, providing 225 the generated new mutation-prompt from the second process 121 to the task-prompt mutation input for performing a further iteration of the first process 111.

[0059] The method 200 includes evaluating 206 the fitness of at least some of the task-prompts in the population. These task-prompts may therefore include task-prompts which were already present in the population, in addition to new task-prompts generated using the first process 111.

[0060] Evaluating the fitness of a particular task-prompt on a training task may comprise providing a training input to a task model to generate a solution (also referred to as an answer) to be evaluated for fitness. When using multiple task-prompts, the fitness evaluation can comprise the following. First, the task model may receive a first training input comprising a first query and a first task-prompt (e.g. associated with a particular individual). The task model can generate a first intermediate output, e.g. conditioned on the first training input. The task model may then receive a second training input comprising the first query, the first taskprompt, the first intermediate output and a second task-prompt (e.g. associated with the same particular individual). The task model can generate a second intermediate output conditioned on the second training input. The task model may then a receive third query (for example a task question), the first training input and the second training input and outputa proposed solution to the problem indicated in the training task. The solution can be evaluated for fitness with reference to a known solution of the training task. A context may also be provided in any ofthe training inputs (e.g. the single training input where a single task-prompt is used, or in one or more of the first, second or additional training inputs where multiple task-prompts are used). The intermediate output may provide a reasoning path used by the task model when determining a first solution to the question problem. It will be appreciated that any other fitness evaluation may be used as will be readily apparent to the skilled person. For example, a fitness may be evaluated using a pre-trained reward model. For example, the task-prompt may be provided as an input to a pre-trained neural network that is trained to estimate how well the task-prompt would perform based on previous evaluations. The method 200 may comprise, in at least one evolutionary round, generating the first task-prompt using the first process 111, and generating the particular task-prompt based on the first task-prompt using the first process 111 in the same evolutionary round.

[0061] Evaluating the fitness of one or more task-prompts may be referred to as an evaluation or an evaluation round. Any number of evolutionary rounds (e.g. one or more) may be performed between successive evaluations. As such, the number of evaluation rounds is typically equal to or greater than the number of evolutionary rounds completed.

[0062] In some implementations, evaluating 206 the fitness of at least some of the task-prompts in the population may advantageously be performed in parallel with one or more of the other steps of the method 200. For example, evaluating 206 the fitness of a task-prompt may be performed in parallel with (e.g. concurrently with, using parallel hardware) performing the first process 111 and / or second process 121 and / or replacing 215 the task-prompt in the population (e.g. the task-prompt associated with the task-prompt mutation input used in the first process 111 in the present evolutionary round) with the new task-prompt generated using the first process 111. The advantages of parallel processing, and how they apply to the systems and methods described herein, are described in more detail above.

[0063] Evaluating the fitness of a first of the at least some of the task-prompts in the population may be performed in parallel with evaluating the fitness of a second of the at least some of the task-prompts in the population. For example, the evaluation process can be implemented using parallel hardware comprising a first processor and a second processor, the first and second processor configured to operate in parallel. Evaluating the fitness of the first of the at least some of the task-prompts in the population may be performed using the first processor and evaluating fitness of the second of the at least some of the task-prompts in the population may be performed using the second processor. Any number of parallel processors (e.g. cores) may be used to evaluate a respective one or more task prompts. As such, each evaluation may beperformed on a dedicated processor of a system comprising multiple processors configured to operate in parallel. This can significantly reduce the clock time of evaluating task-prompts, as well as better leveraging the processing capability of parallel hardware to efficiently perform the methods described herein.

[0064] Additionally or alternatively, evaluating 206 the fitness of at least some of the taskprompts in the population can be segmented into sub-processes and the sub-processes can be performed in parallel. For example, in the above-described example wherein the evaluation comprises the sub-processes of providing: a first training input, a second training input, and a third query; these sub-processes can be parallelized, especially when combining the evaluation of a first task-prompt with the evaluation of a second task-prompt. For example, the evaluation of a first task-prompt may be initiated on a first processor so as to generate a first intermediate output associated with the first task-prompt. The first processor may continue to generate, using the first intermediate output, the second intermediate output in parallel to (e.g. simultaneously with) a second processor beginning to evaluate a second task-prompt and outputting a first intermediate output for the second task-prompt. In this way, a sub-process of the evaluation of the first task-prompt (e.g. the generation of the second intermediate output) is performed in parallel with a sub-process of the evaluation of the second task-prompt (e.g. the generation of the first intermediate output). This specific example is provided for illustrative purposes, but it should be understood that other sub-processes of this, and other, evaluation methods may similarly be parallelized.

[0065] The method 200 also includes selecting 207 a task-prompt from the population based on respective fitnesses of the task-prompts. The selected task-prompt can be input into a model (for example the model 140 of Figure 1, alongside a query Q).

[0066] The method 200 may include evaluating a performance of the new task-prompt, and determining, based on the performance, to control the neural network using the new taskprompt. Evaluating the performance of the new task-prompt may comprise providing a training input to a task neural network to generate an intermediate output, the training input comprising a task specification indicated in the training task and a first task-prompt, providing a second training input to the task neural network to generate a proposed solution to the task specification indicated in the training task, the second training input comprising the first training input, the intermediate output and the new task-prompt, and evaluating the solution with reference to a known solution of the training task. The task-specification may be provided by a user input or may be selected from a list based on a problem domain of the task.

[0067] In some implementations, the method 200 can include, for at least some of the rounds of evolution, maintaining a set of one or more individuals having a highest fitness in that round of evolution. The task-prompts of one or more of this set of one or more individuals can be included in the task-prompt mutation input for performing the first process 121. By maintaining a set of individuals having a highest fitness for at least some rounds of evolution (e.g., a list of “elite” task-prompts), a gradient can be provided to the task-prompt mutation model to assist the task-prompt mutation model in better discerning an efficacious mutation strategy. The taskprompts of the set of individuals may be the only task-prompts included in the task-prompt mutation input. Alternatively, the task-prompt input may include the task-prompt of another individual. For example, the task- prompt input may further include the task-prompt of an individual selected through a tournament as described above. The task-prompts of the set of one or more individuals may be included in the task-prompt mutation input in chronological order. For example, the task-prompts in the set may be ordered in accordance with the round of evolution in which they were generated. The task-prompt mutation input may further comprise an order indicator. For example, where the task-prompts of the set are ordered in ascending order (i.e. task-prompts from later rounds of evolution follow task-prompts from earlier rounds of evolution), an order indicator may include the text “GENOTYPES FOUND IN ASCENDING ORDER.”

[0068] Performing the first process 111 may comprise selecting a mutation strategy from a plurality of mutation strategies and selecting a mutation-prompt based on the selected mutation strategy. Selecting the mutation strategy may comprise sampling the mutation strategies with equal probability. The mutation strategies may include one or more of: direct mutation, Estimation of distribution (EDA), Lamarkian, prompt crossover and context shuffling.

[0069] For at least some of the evolutionary rounds, performing the first process 111 may comprise including a list of task-prompts in the task-prompt mutation input, and including in the task-prompt mutation input a mutation-prompt configured to cause the task-prompt mutation model to continue the list with one or more task-prompts. In this way, the task-prompt mutation model can be considered to estimate a distribution represented by the list of taskprompts. For example, the task-prompts in the list may be task-prompts that have a high fitness. The mutation-prompt may be considered to be an Estimation of Distribution Mutation operator. The method may comprise generating the list of task-prompts. Generating the list of taskprompts may comprise filtering the individuals of the population to remove individuals having a task-prompt that is within a threshold similarity of a task-prompt of another individual of thepopulation. The similarity may be any similarity measure as will be known to those skilled in the art. For example, the similarity may be a cosine similarity of the task-prompts within an embedding space, such as a BERT (Bidirectional Encoder Representations from Transformers) embedding space. The threshold may be any appropriate threshold. In some examples, the threshold may be cosine similarity of 0.95. The list of task-prompts may be included in the task-prompt mutation input in a random order. For example, the list of task-prompts may not be included in order of fitness or in order of generation. Alternatively, the list of task-prompts may be included in the task-prompt mutation input in order of fitness. In some examples, providing the list of task-prompts in ascending order of fitness may make the task-prompt mutation model more likely to generate entries that are similar to elements appearing later in the list due to regency effects of the task-prompt mutation model. As such, by including the fittest task-prompts later in the task-prompt mutation input, the task-prompt mutation model may generate new task-prompts that more closely resemble the fittest task-prompts in the list. An order indication may be included in the task-prompt mutation input indicating an order of the list of task-prompts. The order indication may be a false order indication. That is, the order indication may indicate an order that is different to the order in which the list of task-prompts is actually ordered in the task-prompt mutation input. By providing a false order indication, it is possible to reduce a bias or tendency of the task-prompt mutation model to generating a new task-prompt that is too similar to the final entry in the list of task-prompts and thereby reduce premature convergence and improve exploration of possible solutions.

[0070] For at least some of the evolutionary rounds, when performing the first process 111, providing 213 the task-prompt mutation input may comprise including in the task-prompt mutation input at least one reasoning path. The reasoning path may be a reasoning path of the individual. The reasoning path may be a reasoning path that resulted in a correct answer on at least one training example. The task-prompt mutation input can also include an instruction for the task-prompt mutation model 110 to generate a task-prompt that led to the reasoning path. That is, providing the task-prompt mutation input comprising a mutation-prompt may comprise including in the task-prompt mutation input a mutation-prompt configured to instruct the taskprompt mutation model to generate a task-prompt that led to the reasoning path. This may be considered to be an example of Lamarckian mutation, instructing the task-prompt mutation model to reverse-engineer a task-prompt from a given reasoning path. This type of mutationoperation may be particularly advantageous when there is insufficient information in the problem description, e.g., the problem description is absent or misleading. The reasoning pathmay be part of a context specifically associated with one or more individuals or may be a general context that is not associated with particular individuals. In some examples, the reasoning path may be provided without mutation. In some examples, the reasoning path itself may be mutated, for example by processing an initial reasoning path with a neural network. For example, the initial reasoning path may be mutated based upon a reasoning path mutationprompt. A reasoning path may be referred to as a context.

[0071] A context buffer may be maintained, during the evolutionary process, by adding to the context buffer reasoning paths that led to correct answers during evaluation of task-prompts. The method 200 may include, for at least one evolutionary round, determining whether to perform a context shuffling operation in which a reasoning path in the context buffer is replaced with another reasoning path. The another reasoning path may be sampled from a set of reasoning paths (for example associated with individuals of the population). The determining whether to perform the context shuffling operation may be performed only when a capacity threshold has been reached. For example, the determining may be performed only when the context buffer is full. The determining may be a done probabilistically with a preset probability. For example, the probability may be inversely proportional to the length of the context buffer. The method may include, for at least on evolutionary round, determining whether to replace the entire context buffer with other reasoning paths

[0072] In some implementations the method 200 may comprise, for at least one evolutionary round, determining whether to perform a crossover operation in which a task-prompt of one individual in the population is replaced with a task-prompt of another individual in the population. Determining whether to perform the crossover operation may be done probabilistically with a preset probability, such as 10%. The task-prompt of the other individual may be selected by fitness proportionate selection. The determination may be performed as part of the first process 111 and the individual in which the task-prompt is replaced may be the new task-prompt that has been generated by the first process 111 or may be the task-prompt of the winner itself.

[0073] The method 200 may further include, as part of the evolutionary process, a third process. The third process comprises providing a task-prompt generation input to a task-prompt generation model, the task-prompt generation input comprising an initial problem description and a fixed general prompt. Performing the evolutionary process may comprise, for at least some of the evolutionary rounds, performing the third process to generate new task-prompts in the population 130. The third process provides a zero-order prompt generation process thatdoes not depend on any previously generated task-prompts or mutation-prompts. When the same fixed general prompt is used in every performance of the third process, the third process will sample from the same distribution every time. This can be advantageous when evolution of task-prompts diverges by generating task-prompts that are closely related to the initial problem description. The fixed general prompt may be configured to cause the model to output a plurality of task-prompts and the third process may further comprise selecting a task-prompt from the plurality of task-prompts. For example, the general prompt may be “A list of 100 hints:” or any other prompt configured to instruct the task-prompt generation model to output a plurality of task-prompts. Selecting a task-prompt may comprise sampling from the plurality of task-prompts or selecting a predetermined task-prompt from the plurality (e.g. the 1st, 2nd, 3rd, etc.). Example problem descriptions may include, for example, “solve this math word problem” or “determine whether a text contains hate speech”. Problem descriptions may be associated with a particular domain, for example a mathematical domain, an ethos or ethical domain, a commonsense reasoning domain, an instruction domain.

[0074] The evolutionary process may further use a fourth process. The fourth process comprises providing a mutation-prompt generation input to a mutation-prompt generation model, the mutation-prompt generation input comprising an initial problem description and a thinking style.

[0075] In some example implementations, the techniques described herein may not necessarily involve an evolutionary process. For example, a computer-implemented method for generating a task-prompt for inclusion in a neural network input for controlling a neural network to perform a task may comprise receiving a task specification, determining, based on the task specification, an initial task-prompt, providing a mutation-prompt mutation input as input to a mutation-prompt mutation neural network to generate a mutation-prompt, the mutation-prompt mutation input comprising an initial mutation-prompt and a hypermutation-prompt comprising an instruction to modify the mutation-prompt, and providing the mutation-prompt and the initial task-prompt to a task-prompt mutation neural network in a task-prompt mutation input to generate a new task-prompt.

[0076] Figure 3 depicts an initialization process 300 for populating a population, for example the population 130 described above with reference to Figure 1. The initialization process 400 may be run, for example, prior to performing the method 200 of Figure 2. An initialization process 300 may also be run during an evolutionary process to add new individuals to thepopulation, for example during the evolutionary process comprising the first process 111 and second process 121 of the method 200 of Figure 2.

[0077] The initialization process 300 includes selecting 302 an initial mutation-prompt from a set of mutation-prompts.

[0078] The initialization process 300 further includes including 304 the initial mutationprompt in a prompt initialization input to a prompt initialization model. The prompt initialization model may the same as the task-prompt mutation model and / or the mutationprompt mutation model. Alternatively, the prompt initialization model may be a different model. For example, the prompt initialization model may be a model specifically used specifically for an initialization process. The prompt initialization input may be referred to as an initial task-prompt input. The prompt initialization model may be a neural network (e.g. a prompt initialization neural network). The prompt initialization model may be referred to as a task-prompt initialization model.

[0079] The initialization process 300 includes generating an initial task-prompt using the prompt initialization model. That is, the prompt initialization model receives the prompt initialization input including the initial mutation-prompt and generates, as an output, an initial task-prompt.

[0080] The initial task-prompt may be one of a plurality of task-prompts provided in the taskprompt mutation input. The plurality of task-prompts may comprise a list of task-prompts, and the mutation-prompt may be configured to cause the task-prompt mutation neural network to continue the list with one or more task-prompts. The task-prompts of the plurality of taskprompts may be included in the task-prompt mutation input in an order, for example chronological order, a random order, or in order of performance (fitness). An order indication may also be included in the task-prompt mutation input indicating an order of the plurality of task-prompts. The order indication may be a false order indication as described above.

[0081] The initialization process 300 further includes adding an individual comprising the initial task-prompt to the population.

[0082] The initialization process 300 may be run on a separate computer system to either or both of the first process 111 and the second process 121. The prompt initialization model may the same as the task-prompt mutation model 110 and / or the mutation-prompt mutation model 120. Alternatively, the prompt initialization model may be a different model. For example, the prompt initialization model may be a model specifically used specifically for an initializationprocess. The initialization process may be run in parallel across a number of processors, thereby enabling an initial population of individuals to be generated quickly and efficiently.

[0083] The prompt initialization input provided to the prompt initialization model may also include additional modifiers. Initializing the population may comprise selecting a modifier from a set of modifiers and including the modifier in the prompt initialization input to the prompt initialization model to generate the initial task-prompt. The modifiers may also be referred to as thinking styles and may represent particular ways in which initialization model may approach generation of the task-prompt. Example thinking style include, but are not limited, to the following: “Let’s think step by step”, “Ignoring the current best solution, create an entirely new solution to the problem”, “Use risk analysis: evaluate potential risks, uncertainties and trade-offs associated with different solutions or approaches to a problem”, “Does the problem involve a physical constraint, such as limited resources, infrastructure, or space?”.

[0084] The initial task-prompt mutation input may include at least one example reasoning path (e.g. a context). The mutation-prompt may be configured to instruct the task-prompt mutation neural network to generate a task-prompt that led to the at least one example reasoning path. The method may include generating the at least one example reasoning path. The generating may comprise mutating an initial at least one example reasoning path based on a reasoning path mutation-prompt.

[0085] The initialization process 300 may further comprise maintaining a context buffer comprising one or more reasoning paths for inclusion in the task-prompt mutation input. There may be a determination of whether to perform a context shuffling operation in which a reasoning path in the context buffer is replaced with another reasoning path. Determining whether to replace a reasoning path may include determining whether to replace the entire context buffer with other reasoning paths.

[0086] Initializing the population may additionally or alternatively comprise including a task specification / problem description in the prompt initialization input to the prompt initialization model to generate the initial task-prompt. The problem description may be domain-specific. A number of example domains and types of problem descriptions is described in more detail below. The task-specification may be provided by a user input or may be selected from a list based on a problem domain of the task.

[0087] Determining the initial task-prompt may comprise including the task specification in a prompt initialization input to a prompt initialization neural network to generate the initial task-prompt. The prompt initialization neural network may the same as the task-prompt mutation neural network and / or the mutation-prompt mutation neural network and may be the same as the neural networks described above. The initial mutation-prompt may be included in the prompt-initialization input. The initial mutation-prompt may be selected from a set of mutation-prompts. A modifier may also be included in the prompt initialization input. The modifier may represent a thinking style. The modifier may be selected from a set of modifiers. Modifiers and thinking styles are describe in more detail above.

[0088] Given that the initialization process 300 may be run prior to, or during, the performance of the prompt-evolution process 200 as described above with reference to Figure 2, any of the essential or optional sub-processes and features of the prompt-evolution process 200 may be used in combination with the initialization process 300.

[0089] Figure 4 is a flow diagram of an example method 400 for generating a task-prompt. For convenience, the method 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a prompt evolution system, e.g., the prompt evolution system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the method 400. The generated task-prompt may be for inclusion in a model input for controlling a model to perform a task. The model may be, for example, the model 140 of Figure 1, e.g. configured to receive a query Q and a task-prompt and output an answer A. The method 400 may be referred to as a prompt evolution method or process.

[0090] The method 400 includes receiving 402 a task specification. The task specification may be provided by a user input or may be selected from a list based on a problem domain of the task.

[0091] The method 400 further includes determining 404, based on the task specification, an initial task-prompt. Determining the initial task-prompt may comprise including the task specification in a prompt initialization input to a prompt initialization neural network to generate the initial task-prompt. For example, determining the initial task-prompt may include using the initialization method 300 described above with reference to Figure 3. The initial mutation-prompt (e.g. used in the prompt initialization method) may be included in the promptinitialization input. The prompt initialization input may additionally or alternatively comprise a modifier. Modifiers are described in more detail above. More than one initial task-prompt may be determined for use in the method 400.

[0092] The method 400 further includes generating 406 a mutation-prompt. Generating 406 the mutation-prompt includes providing a mutation-prompt mutation input as input to amutation-prompt mutation model. The mutation-prompt mutation model may be, for example, the mutation-prompt mutation model 120 of Figure 1 for the purpose of generating a mutationprompt. The mutation-prompt mutation input comprises an initial mutation-prompt and a hypermutation-prompt comprising an instruction to modify the mutation-prompt. The initial mutation-prompt may be included in the prompt-initialization input. The prompt-initialization input may further include a modifier. Modifiers are described in more detail above.

[0093] The method 400 further includes providing 408 the mutation-prompt and the initial task-prompt to a task-prompt mutation model in a task-prompt mutation input. The task-prompt mutation model may be, for example, the task-prompt mutation model 110 of Figure 1, e.g. configured to receive a task-prompt mutation input and generate a new task-prompt. More than one initial task-prompt may be provided in the task-prompt mutation input. That is, the initial task-prompt may be one of a plurality of task-prompts provided in the task-prompt mutation input. The plurality of task-prompts can include a list of task-prompts, and the mutation-prompt may be configured to cause the task-prompt mutation model to continue the list with one or more task-prompts. The task-prompts of the plurality of task-prompts can be included in the task-prompt mutation input in chronological order, random order, in order of performance, or in any other order. The task-prompt mutation input can include an order indication indicating an order of the plurality of task-prompts.

[0094] The task-prompt mutation input may further include at least one reasoning path. The task-prompt mutation input may further include an instruction for the task-prompt mutation neural network to generate a task-prompt that led to the reasoning path. The mutation-prompt may be configured to instruct the task-prompt mutation neural network to generate a taskprompt that led to the at least one example reasoning path. The reasoning path may be generated as part of the method 400. Generating the reasoning path may include mutating an initial at least one example reasoning path based on a reasoning path mutation-prompt.

[0095] The method 400 further includes generating 410 a new task-prompt. Generating the new task-prompt may include mutating the initial task-prompt based on the mutation-prompt.

[0096] Optionally, the method 400 may further comprise maintaining a context buffer comprising one or more reasoning paths for inclusion in the task-prompt mutation input. There may be a determination of whether to perform a context shuffling operation in which a reasoning path in the context buffer is replaced with another reasoning path. Determining whether to replace a reasoning path may include determining whether to replace the entire context buffer with other reasoning paths.

[0097] Optionally, the method 400 may include performing the task by controlling a model (e.g. the model 140 of Figure 1) using the new task-prompt. The method 400 may include evaluating a performance of the new task-prompt (e.g. generated in step 410). The method 400 may further include determining, based on the evaluated performance, to control the model using the new task-prompt.

[0098] Evaluating the performance of the new task-prompt may comprise providing a first training input to a task neural network to generate an intermediate output, the first training input comprising a task specification indicated in the training task and a first task-prompt, providing a second training input to the task neural network to generate a proposed solution to the task specification indicated in the training task, the second training input comprising the first training input, the intermediate output and the new task-prompt, and evaluating the solution with reference to a known solution of the training task.

[0099] The method 400 may further comprise receiving a user input comprising a problem description, modifying the user input to include the selected task-prompt, and providing the modified user input as input to a task model to generate a task model output.

[0100] Once the selected task-prompt has been generated, the task-prompt may be stored for later use. That is, the use of the selected task-prompt need not take place at the same time or on the same computer system as the generation of the selected task-prompt. The selected taskprompt may be stored along with other generated task-prompts for later use. For example, a plurality of task-prompts may be generated using the methods described herein, each of the plurality of task-prompts being specific to a particular problem domain. At a later time, when receiving a user input comprising a problem description, the problem description may be processed to determine which of the plurality of task-prompts will be most effective. For example, a problem domain of the problem description may be determined and matched with a problem domain of one of the plurality of task-prompts. In this way, a library of task-prompts for different problem domains may be generated and maintained for use across a number of problem domains.

[0101] Figure 5 is a flow diagram of a method 500 for performing a task. The task may be a task which a model is configured to perform. The model may be, for example, the model 140 shown in Figure 1, which is configured to receive a query Q and a task-prompt 131 and output an answer A so as to perform a task. For convenience, the method 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a prompt evolution system, e.g., the prompt evolution system 100 of FIG. 1,appropriately programmed in accordance with this specification, can perform the method 500. The method 500 may be referred to as a prompt evolution method or process.

[0102] The method 500 includes receiving 502 a task specification. The task specification may be provided by a user input or may be selected from a list based on a problem domain of the task.

[0103] The method 500 includes determining 504, based on the task specification, an initial task-prompt. Determining the initial task-prompt may comprise generating the initial taskprompt based on the task specification. The initial task-prompt may be generated further based on an initial mutation-prompt, for example the same initial mutation-prompt as that received in the subsequent step 506 of the method 500. The initial task-prompt may be generated further based on a prompt modifier. Modifiers are described in more detail above. The initial taskprompt may comprise at least one reasoning path.

[0104] The method 500 further includes receiving 506 an initial mutation-prompt. The initial mutation-prompt may be obtained by the methods described above, for example using the initialization method 300 described above with reference to Figure 3, for example using a taskprompt mutation model. Where the initial task-prompt comprises at least one reasoning path, the mutation-prompt may be further configured to instruct a task-prompt mutation model to generate a task-prompt that led to the at least one example reasoning path. Optionally, the method 500 further comprises generating the at least one example reasoning path. Generating the at least one example reasoning path may comprise mutating an initial at least one reasoning path based on a reasoning path mutation-prompt.

[0105] The method 500 further includes generating 508 a mutation-prompt (e.g. a new mutation-prompt). Generating the mutation-prompt comprises mutating the initial mutationpromptbased on a hypermutation-prompt. The hypermutation-prompt comprises an instruction to modify the mutation-prompt. Generating the mutation-prompt may include providing the initial mutation-prompt mutation input as input to a mutation-prompt mutation model. The mutation-prompt model may be, for example, the mutation-prompt mutation model 120 of Figure 1. The initial mutation-prompt may be included in a prompt-initialization input for input into the mutation-prompt mutation model. The prompt-initialization input may further include a modifier. Modifiers are described in more detail above.

[0106] The method 500 further includes generating 510 a new task-prompt. Generating the new task-prompt comprises mutating the initial task-prompt based on the mutation-prompt. The generation of the new task-prompt may be further based on at least one reasoning path ina context buffer. Context buffers and their maintenance are described in more detail above. The method 500 may further comprise determining whether to perform a context shuffling operation in which a reasoning path in the context buffer is replaced with another reasoning path. In some instances, a determination may be made whether to replace the entire context buffer with other reasoning paths.

[0107] Optionally, the method 500 may further comprise performing 512 the task. Performing the task may include controlling a model, for example the model 140 shown in Figure 1, using the new task-prompt. The method 500 may further include evaluating a performance of the new task-prompt (e.g. generated in step 510). The method 500 may further include determining, based on the evaluated performance, to control the model using the new task-prompt. Evaluating the performance of the new task-prompt may comprise generating a training input comprising a task specification indicated in the training task and a first task-prompt, generating an intermediate output based on the training input, generating a second training input comprising the first training input, the intermediate output and the new task-prompt, generating a solution the task specification based on the second training input and evaluating the solution with reference to a known solution of the training task.

[0108] Figure 6 depicts the fitness of a population of task-prompts over a number of fitness evaluations, for example as described with reference to step 206 of the method 200 of Figure 2. The task-prompts have been evolved using the methods described above, for example using the method 200 according to Figure 2 using the prompt evolution system 100 of Figure 1. Each individual comprises two task-prompts (a pair of task-prompts). The two task-prompts are used in a prompt strategy which sequentially applies two task-prompts to the task at hand as described in more detail above. In this example, the population comprises 20 individuals.

[0109] Each dot represents a pair of task-prompts (e.g. associated with an individual of the population) and their associated fitness, for a particular evaluation. Due to the large size of the dataset, not every possible task-prompt may be depicted and / or task-prompts may not be depicted for every evolutionary round. Rather, an example subset of the task-prompts are shown for illustrative purposes. The fitness is scored for a particular task. In this example, the task is a ‘word in context’ task. In the word in context task, the model is provided with two sentences and a homograph word, and must determine whether the homograph word has been used with the same meaning in both sentences. The evaluations are performed as a binary tournament wherein two individuals in the population are evaluated. To evaluate the fitness of each evolved task-prompt pair, a batch of query (Q) and answer (A) pairs are sampled from atraining data set. The fitness is scored between 0 and 100. Over the 2000 evaluations, the fitness of task-prompts increases considerably.

[0110] A maximum fitness line 61 shows the maximum task-prompt fitness at each evaluation. The value of the maximum fitness is initially around 21 in the first evaluation. After 250 evaluations, the maximum fitness of the task-prompts increases to approximately 50. After 500 evaluations, the mean of the task-prompts increases to approximately 60. After 1000 evaluations, the mean of the task-prompts increases to approximately 72. After 2000 evaluations, the mean of the task-prompts increases to approximately 82. The trend in the maximum fitness line 61 suggests a trend of improvement in maximum fitness as the evaluation number increases. This indicates that, by continuing to evolve the task-prompts and continuing to evaluate beyond 2000 evaluations, the maximum fitness may be increased further.

[0111] In Figure 6, the content of seven pairs of task-prompts 131-1, 131-2, 131-3, 131-4, 131- 5, 131-6, 131-7 with the highest fitness in various evaluations is shown. These task-prompts 131-1, 131-2, 131-3, 131-4, 131-5, 131-6, 131-7 with the highest fitness can be referred to as elite prompts.

[0112] A mean fitness line 62 shows the mean task-prompt fitness at each evaluation. In the first evaluation, the mean fitness of the task-prompts is less than 10. After 250 evaluations, the mean fitness of the task-prompts increases to approximately 15. After 500 evaluations, the mean of the task-prompts increases to approximately 35. After 1000 evaluations, the mean of the task-prompts increases to approximately 50. After 2000 evaluations, the mean of the taskprompts increases to approximately 65. The trend in the mean fitness line 62 suggests a trend of improvement in mean fitness as the number of mutations and therefore number of evaluations increases. As such, by continuing to evolve the task-prompts and continuing to evaluate beyond 2000 evaluations, the mean fitness may be increased further.

[0113] The example depicted in Figure 6 is for illustrative purposes and, in other examples, any number of evaluations may be used. An evaluation may be performed after every evolutionary round or after more than one evolutionary round. As such, the number of evaluations may be the same as or greater than the number of evolutionary rounds. Furthermore, different fitness scoring ranges and metrics may be used in other examples.

[0114] The accuracy of a model which uses task-prompts evolved according to the methods outperforms many alternative prompt-strategies on a range of commonly used reasoning benchmarks. In an example implementation, the prompt evolution system 100 is implemented with a population 130 comprising 50 individuals, and the method 200 for generating newprompts is performed for 30 evaluations. The population 130 is updated by competing all of the individuals of the population 130 against each other in an evaluation. In this example, the task-prompt mutation model 110 and mutation-prompt mutation model 120 comprise an LLM, and in particular the pathways language model (PaLM) version PaLM 2-L.

[0115] The performance of the resulting task-prompts demonstrate an improved accuracy compared to alternative prompt-strategies for arithmetic reasoning tasks, commonsense reasoning tasks, instruction induction tasks and hate speech classification tasks. The evolved task-prompts enable a zero-shot accuracy of 99.7%, 96.4%, 87.8% and 90.2% for the four different arithmetic reasoning tasks: MultiArith (Roy & Roth, arXiv: 1608.01413), SingleEq (Koncel-Kedziorski et al., D01: 10.1162 / tacl_a_00160), AddSub (Hosseini et al., DOI: 10.3115 / vl / D14-1058) and SVAMP (Patel et al., DOI:10.18653 / vl / 2021.naacl- main.168). This compares with zero-shot accuracies of 97.7%, 94.7%, 74.4% and 86.3%, for each of MultiArith, SingleEq, AddSub and SVAMP, respectively, associated with the best performances for an alternative prompt-strategy, Pl an-and- Solve (Wang et al., DOI:10.18653 / vl / 2023.acl-long.l47), implemented with PaLM 2-L (Anil et al., 2023, PaLM Technical Report) as the underlying LLM. Similarly, the evolved task-prompts enable a zeroshot accuracy of 71.8% and 85.4% for two different commonsense reasoning tasks: CommonsenseQA (Talmor et al., DOL 10.18653 / vl / N19-1421) and StrategyQA (Geva et al., DOI: 10.1162 / tacl_a_00370) compared with best accuracies of 50.1% and 77.9% for CommonsenseQA and StrategyQA, respectively, achieved using Plan-and-Solve.

[0116] A few-shot accuracy of up to 100.0% is achieved using the evolved task-prompts for arithmetic reasoning tasks, in comparison to a best performance of 95.5% achieved for an alternative prompt-strategy, in particular Chain-of-Thought prompting using text-davinci-003 as an underlying LLM (Wei et al., arXiv: 2201.11903).

[0117] When applying the methods described herein to alternative tasks, an improved accuracy compared to alternative prompt- strategies is achieved. For example, in a hate speech classification task (using the ETHOS dataset, Mollas et al., DOI: 10.1007 / s40747-021-00608- 2) an accuracy of 89% is achieved compared to an accuracy of 80% for hand-designed prompts. In another example, in an instruction induction task the task-prompts evolved using the methods described herein match or surpass alternative prompt-strategies in 21 out of 24 instruction induction tasks. This demonstrates that the methods described herein are capable of adapting task-prompts to a specific domain and / or specific task with high accuracy. Furthermore, the fitness of task-prompts generated using the methods described hereincontinue to increase as the number of evaluations increases. This is in contrast to other prompt strategies which do not necessarily show such a continual increase in fitness.

[0118] As described above, evaluating a fitness of task-prompts can include evaluating their fitness on one or more training tasks. The one or more training tasks may be associated with a task that the model (e.g. the model 140 of Figure 1) is configured to perform.

[0119] Below, a number of example training tasks are described. Reference is made to a problem description, for example that an aspect of the task is indicated in or specified in a problem description. Any of the methods described above (e.g. methods 200, 300, 400, 500) may include receiving a user input comprising a problem description, modifying the user input to include the selected task-prompt, and providing the modified user input as input to a task model (e.g. task neural network) to generate a task model output (e.g. task model neural network output).

[0120] The one or more training tasks may be computer-security related tasks. For example, the training examples may involve generating computer program code for execution to address a security breach indicated in a problem description. Other example tasks are described below. Where the training tasks are computer-security related tasks, the user input may indicate a security issue in a computing device, computer-network or in computer software code and the task model output comprises computer program code for execution by the device or computer network.

[0121] The one or more training tasks may relate to control of an agent to one or more perform task. For example, the training examples may involve generating instructions to control an agent to perform a task specified in a problem description. The problem description may include sensor data, such as image data, or other sensor data.

[0122] Where the training tasks relate to control of an agent, the user input may comprise input data derived from one or more sensors and indicating one or more properties of one or more physical objects in a real -word environment. The user input may further comprise a request to output one or more instructions to complete a task including an action on or using the one or more physical objects. The task model output may comprise one or more instructions for execution by a real-world agent interacting with the environment. The method may include controlling the real-world agent to perform action using the task model output.

[0123] The one or more training tasks may relate to processing of sensor data to identify conditions present in the sensor data. For example, the one or more training tasks may relate to processing images to identify properties of the image or whatever is depicted in the image. Forexample, the training tasks may relate to identification of objects depicted in images. That is, the condition may be the presence or absence of an object, person or thing in the real -world environment. As another example, the training tasks may relate to processing sensor data relating to a system to identify a malfunction or potential malfunction in the system. That is the condition may be the presence of a malfunction or may be one or more conditions that could lead to a malfunction.

[0124] Where the training tasks relate to processing of sensor data, the user input may comprise input data derived from one or more sensors and indicating one or more properties of a real- world environment. The user input may further comprise a request to identify one or more conditions of the real-world environment based on the input data. The task model output may comprise an indication of whether the one or more conditions are present in the real-world environment. The method may include controlling an agent in response to the task model output indicating that the one or more condition is, or is not, present in the environment.

[0125] The one or more training tasks may relate to the generation and / or debugging of computer-program code. For example, a training task may comprise a description of a desired functionality and the solution of the training task may comprise computer program code that causes one or more computers to perform the desired functionality. As another example, the training task may comprise computer-program code that does not properly perform a desired functionality and the solution of the training task may be an indication that the computerprogram code will not perform the desired functionality and / or an indication of one or more portions of the computer-program code that is incorrect and / or an indication of alternative computer program code that would fix one or more portions so that the fixed computer program code would properly perform the desired functionality.

[0126] The methods described herein can include receiving a user input. The user input may comprise, for example, a problem description. The corresponding method further comprises modifying the user input to include the selected task-prompt, and providing the modified user input as input to a task model to generate a task model output.

[0127] In instances where a user input is received, the user input may indicate a portion of computer program code and a request to identify the presence of a bug in the computer program code. The task model output may comprise an indication of a bug in the computer program code. The method may further comprise taking an action, such as preventing use of the computer program code in response to the task model output indicating the presence of the bug.

[0128] Any of the models described herein (e.g., the task-prompt mutation model, the mutation-prompt mutation model, the task-prompt generation model, the mutation-prompt generation model, the task-prompt initialization model, the task model) may be sequence models. As will be understood by the skilled person, a sequence model is a machine learning model that receive data indicative of an input sequence and / or outputs sequences of data. For example, the models may comprise one or more self-attention layers, such as Transformerbased so called Large Language Models (LLMs) e.g. with more than IO10learnable parameters, but applications of the described techniques are not restricted to such models. The first and / or second computer-implemented agents may be language generation neural network. Language generation neural networks typically comprises a sequence-to-sequence model that receives an input sequence of natural language tokens and generates an output sequence of natural language tokens. Typically a natural language token defines a word or wordpiece (e.g. a word segment or morpheme), but it may also define a letter, or multiple words. The tokens may include tokens representing punctuation. In some implementations an output sequence of natural language tokens is generated a word or wordpiece at a time. In general any language generation neural network may be used, e.g. an auto-regressive language generation neural network, or a language generation neural network that does not rely on an auto-regressive model such as a recurrent language generation neural network or a denoising auto-encoder based language model (arXiv:2112.06749). As an example, a language generation neural network as described herein can be a transformer-based language model neural network, in particular an autoregressive transformer-based language model neural network. A transformer neural network may be characterized by having a succession of self-attention neural network layers. The first and / or second computer-implemented agents may be language models. A language model is similar to a language generation model but may process an input sequence of natural language tokens to generate a vector or scalar output rather than generating an output sequence of natural language tokens.

[0129] One or more of models may be a multimodal model. A multimodal model may have any appropriate architecture, such as an attention-based neural network architecture, e.g., a transformer architecture, a convolutional architecture, a fully-connected architecture, or any other appropriate neural network architecture. The model can include any appropriate types of neural network layers, e.g., convolutional layers, attention layers, fully connected layers, recurrent layers, and so forth, in any appropriate numbers, e.g., 10 layers, 100 layers, or 1000 layers, and connected in any appropriate configuration, e.g., as a linear sequence of layers oras a directed graph of layers. Some implementations of the models described herein can be described as “large” multimodal models, e.g. with more than IO10learnable parameters, but applications of the described techniques are not restricted to such models.

[0130] A multimodal machine learning model has a multimodal input configured to receive a first multimodal input and a second multimodal input. As used herein a “modality” refers to a type of data, and thus a multimodal machine learning model is one that can process multiple different types of data. For example, a first multimodal input may comprise a text input to receive a sequence of text. A second multimodal input may be configured to receive a different type of input data, and in some implementations second multimodal input comprises a visual input to receive an image or video. Alternatively the second multimodal input may be configured to receive, e.g., audio data representing values of an audio waveform, e.g. instantaneous amplitude data or time-frequency domain data; or data representing observations (not necessarily visual) of an environment with which an agent controlled by the multimodal machine learning model interacts. In some implementations there may be more than two different multimodal inputs, each configured to receive a different type of data.

[0131] A multimodal machine learning model is configured to jointly process an encoded version of the text and an encoded version of the second multimodal input, e.g. of the image or video, to generate a model output that defines a result of a machine learning task. A few examples of machine learning tasks that can be performed by a multimodal machine learning model are described later.

[0132] Text received by a text input of a multimodal model may comprise text in one or more natural languages, or text in a computer language, or both. The computer language may be any formal language used to communicate with a computer, e.g. a markup language, or a command or configuration language, or a data exchange language such as JSON, or a programming language. The text may be received, e.g., as a series of encoded characters, e.g. UTF-8 encoded characters; such “characters” can include Chinese and other similar characters, as well as logograms, syllabograms and the like.

[0133] In some implementations, but not necessarily, a multimodal machine learning model includes a text encoder that processes the sequence of text to represent the text as a series of text tokens from a vocabulary of text tokens, e.g. that each represent words, wordpieces or characters in a natural or computer language.

[0134] In some implementations a second multimodal input of a multimodal model comprises an image or video. In these implementations the second multimodal input may comprise imagedata defining color or intensity values for pixels of a still or moving image in one, two, or three dimensions. As used herein “image” includes a LIDAR point cloud, and the image data may also or instead define the locations of points of a still or moving point cloud. As another example, an image or video received by the second multimodal input may comprise a neural 3D representation, e.g. that represents a 3D scene as a set of latent feature vectors, e.g. a neural radiance field representation.

[0135] In some implementations, but not necessarily, a multimodal machine learning model includes a second multimodal input encoder that processes the second multimodal input, e.g. using one or more convolutional, attention, fully connected, or recurrent layers, to generate the encoded version of the second multimodal input. In general the encoder may implement any form of encoding appropriate for the type of data to be encoded. Merely as an example, where the second multimodal input comprises an image or video this may be encoded, e.g., as features for each of a set of patches that tile the image, or as a sequence of visual tokens selected from a vocabulary of visual tokens, or as a representation of distinct objects in the visual input. Such visual tokens may, but need not be, interleaved with text tokens processed by the model.

[0136] When implementing the task model, the model output may comprise any form of output appropriate to the machine learning task performed by the multimodal machine learning model. In some implementations the model output comprises text in a natural or computer language that defines a result of the task, e.g. for tasks such as image captioning, visual question answering, or object detection or instance segmentation. Also or instead the model output may comprise data defining an image, video or audio object, e.g. in a generative task; or the model output may comprise non-textual action selection data for selecting an action to be performed by an agent controlled by the model. As another example the model output may also or instead define an intermediate step to be performed during the task, e.g. a call to a software API for a software tool that is used when performing the task; the multimodal input may then receive an output from the software tool that is used to generate a final model output that performs the task. A few particular examples of model output are given later.

[0137] implementations a multimodal machine learning model includes one or more joint processing neural network layers that jointly process data from the text input and the second multimodal input to generate the model output. As some examples, data from the text input can be processed conditioned on the encoded version of the second multimodal input, or vice- versa; or one or more cross-attention layers can be used to enable data from data from the text input to attend to data from the second multimodal input, or vice-versa, to provide fusedmultimodal input data; or the multimodal input data may be fused in some other way, e.g. by concatenation; or a series of text tokens and tokens derived from the second multimodal input, e.g. visual tokens, can be processed by a shared set of processing layers, e.g. configured as a transformer neural network encoder and / or decoder.

[0138] Merely as one particular example the joint processing neural network layers may comprise a transformer neural network, i.e. a neural network with a transformer architecture, e.g. a transformer encoder and / or transformer decoder neural network. In general such a transformer neural network may be characterized by having a succession of self-attention neural network layers. A self-attention neural network layer has an attention layer input for each element of the input and is configured to apply an attention mechanism over the attention layer input to generate an attention layer output for each element of the input. There are many different attention mechanisms that may be used.

[0139] Some example multimodal machine learning models with which the techniques described herein may be used include: Flamingo (Alayrac et al. arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al. arXiv:2209.06794); and PaLLX (Chen et al. arXiv:2305.18565).

[0140] In some implementations, the second multimodal input may include an observation characterizing an environment of an agent performing a task, e.g. a mechanical agent or software agent. The observation may characterize the environment at a particular time step and the model output may define one or more actions to be performed by the agent at the time step. For example each action may be expressed as a sequence of text, e.g. as one or more characters such as letters and numbers, that represents the action, or as text that defines a low- level “skill” from a set of skills; or the model output may, e.g., define parameters of a probability distribution from which an action is selected. Optionally text received by text input may include text describing the task to be performed. Optionally the text input may include a description of one or more actions performed at a preceding time step. Where the agent is a software agent the model output may comprise a text output for calling a software API at a time step, and the model input at a subsequent time step, e.g. the text input, may comprise a response from the software agent, e.g. from the API.

[0141] Some examples of multimodal machine learning models controlling an agent, and with which the techniques described herein may be used, are described in: PaLM-E (Driess et al. arXiv:2303.03378); RT-1 (Brohan et al. arXiv:2212.06817); and RT-2 (Brohan et al. arXiv:2307.15818).

[0142] As previously described, in some implementations a multimodal machine learning model can have one or more further multimodal inputs. For example the model may include an audio input, or an agent action input to receive agent action data representing an action of an agent performing a task in an environment. Data received in this way may be jointly processed with data from the text input and second multimodal input to generate the model output.

[0143] A multimodal machine learning model has a multimodal input and can, in implementations, perform a range of different tasks. However in implementations not every task that the model performs requires a multimodal input, e.g. a task to generate an image from a text description of the image, or an image captioning task. In each case, however, by using the described techniques to evolve task-prompts, operation of the multimodal model may be improved.

[0144] As described above, the method may be used to generate a task-prompt for use in generating instructions to control an agent to perform an action to complete a task in an environment. In particular, the system can be used to evolve a task-prompt that facilitates a task model to generate an output that can be used to control an agent to perform an action.

[0145] In some implementations, the environment is a real-world environment, the agent is a mechanical agent interacting with the real -world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot interacting with the environment to accomplish a specific task, e.g., to locate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.

[0146] In these implementations, the training tasks and / or the user input to a task model may include an observation, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator. For example in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose ofone or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and / or relative observations. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and / or image or video data for example from a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

[0147] In these implementations, the answers of the training examples, and / or the output of a task model, may include actions to control an agent to perform the task. The actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements, e.g., steering control elements of the vehicle, or higher-level control commands. The control signals can include for example, position, velocity, or force / torque / accel eration data for one or more joints of a robot or parts of another mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land or air or sea vehicle the control signals may define actions to control navigation, e.g., steering, and movement, e.g., braking and / or acceleration of the vehicle.

[0148] In some implementations the environment is a simulation of the above-described real- world environment, and the agent is implemented as one or more computers interacting with the simulated environment.

[0149] In some implementations the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product. As used herein a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material, e.g., to remove pollutants, to generate a cleaned or recycled product. The manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g., robots, for processing solid or other materials. The manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g., via pipes or mechanical conveyance. As used herein manufacture of a product also includes manufacture of a food product by a kitchen robot.

[0150] The agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the product. That is, the agent may comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing units or machines or to control movement of an intermediate version or component of the product between the manufacturing units or machines.

[0151] As one example, a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof. As another example, a task performed by the agent may comprise a task to control, e.g., minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process.

[0152] The actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment, e.g., between the manufacturing units or machines. In general the actions may be any actions that have an effect on the observed state of the environment, e.g., actions configured to adjust any of the sensed parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. The actions may include actions imposing operating conditions on a manufacturing unit or machine, or actions that result in changes to settings to adjust, control, or switch on or off the operation of a manufacturing unit or machine.

[0153] In general, observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and / or mechanical items of equipment. For example a representation of the state of the environment may be derived from observations made by sensors sensing a state of the manufacturing environment, e.g., sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines. As some examples such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate or flow / movement path of one or more materials; physical or chemical conditions, e.g., a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations;image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor. In the case of a machine such as a robot the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g., data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal, or image or video data for example from a camera or a LIDAR sensor. Sensors such as these may be part of or located separately from the agent in the environment.

[0154] In some implementations the environment is the real-world environment of a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include ancillary control equipment that controls an operating environment of the items of equipment, for example environmental control equipment such as temperature control, e.g., cooling equipment, or air flow control or air conditioning equipment such as a heater, a cooler, a humidifier, or other hardware that modifies a property of air in the real-world environment. The task may comprise a task to control, e.g., minimize, use of a resource, such as a task to control electrical power consumption, or water consumption. The agent may comprise an electronic agent configured to control operation of the items of equipment, or to control operation of the ancillary, e.g., environmental, control equipment.

[0155] In general the actions may be any actions that have an effect on the observed state of the environment, e.g., actions configured to adjust any of the sensed parameters described below. These may include actions to control, or to impose operating conditions on, the items of equipment or the ancillary control equipment, e.g., actions that result in changes to settings to adjust, control, or switch on or off the operation of an item of equipment or an item of ancillary control equipment.

[0156] In general observations of a state of the environment may comprise any electronic signals representing the functioning of the facility or of equipment in the facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a state of a physical environment of the facility or observations made by any sensors sensing a state of one or more of items of equipment or one or more items of ancillary control equipment. These include sensors configured to sense electrical conditions such ascurrent, voltage, power or energy; a temperature of the facility; fluid flow, temperature or pressure within the facility or within a cooling system of the facility; or a physical facility configuration such as whether or not a vent is open.

[0157] In some implementations the environment is the real-world environment of a power generation facility, e.g., a renewable power generation facility such as a solar farm or wind farm. The task may comprise a control task to control power generated by the facility, e.g., to control the delivery of electrical power to a power distribution grid, e.g., to meet demand or to reduce the risk of a mismatch between elements of the grid, or to maximize power generated by the facility. The agent may comprise an electronic agent configured to control the generation of electrical power by the facility or the coupling of generated electrical power into the grid. The actions may comprise actions to control an electrical or mechanical configuration of an electrical power generator such as the electrical or mechanical configuration of one or more renewable power generating elements, e.g., to control a configuration of a wind turbine or of a solar panel or panels or mirror, or the electrical or mechanical configuration of a rotating electrical power generation machine. Mechanical control actions may, for example, comprise actions that control the conversion of an energy input to an electrical energy output, e.g., an efficiency of the conversion or a degree of coupling of the energy input to the electrical energy output. Electrical control actions may, for example, comprise actions that control one or more of a voltage, current, frequency or phase of electrical power generated.

[0158] In general observations of a state of the environment may comprise any electronic signals representing the electrical or mechanical functioning of power generation equipment in the power generation facility. For example, a representation of the state of the environment may be derived from observations made by any sensors sensing a physical or electrical state of equipment in the power generation facility that is generating electrical power, or the physical environment of such equipment, or a condition of ancillary equipment supporting power generation equipment. Such sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power or energy; temperature or cooling of the physical environment; fluid flow; or a physical configuration of the equipment; and observations of an electrical condition of the grid, e.g., from local or remote sensors. Observations of a state of the environment may also comprise one or more predictions regarding future conditions of operation of the power generation equipment such as predictions of future wind levels or solar irradiance or predictions of a future electrical condition of the grid.

[0159] As another example, the environment may be a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals / intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical. As another example, the agent may be a mechanical agent that indirectly performs or controls the protein folding actions, e.g., by controlling chemical synthesis steps selected by the system automatically without human interaction. The observations may comprise direct or indirect observations of a state of the protein or chemical / intermediates / precursors and / or may be derived from simulation. Thus, the system may be used to automatically synthesize a protein with a particular function such as having a binding site shape, e.g. a ligand that binds with sufficient affinity for a biological effect that it can be used as a drug. For example e.g. it may be an agonist or antagonist of a receptor or enzyme; or it may be an antibody configured to bind to an antibody target such as a virus coat protein, or a protein expressed on a cancer cell, e.g. to act as an agonist for a particular receptor or to prevent binding of another ligand and hence prevent activation of a relevant biological pathway.

[0160] In a similar way, the environment may be a drug design environment such that each state is a respective state of a potential pharmaceutically active compound and the agent is a computer system for determining elements of the pharmaceutically active compound and / or a synthetic pathway for the pharmaceutically active compound. The drug / synthesis may be designed based on a reward derived from a target for the drug, for example in simulation. As another example, the agent may be a mechanical agent that performs or controls synthesis of the drug.

[0161] In some further applications, the environment is a real-world environment and the agent manages distribution of tasks across computing resources, e.g., on a mobile device and / or in a data center. In these implementations, the actions may include assigning tasks to particular computing resources.

[0162] As a further example, the actions may include generating or presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, andthe task may be to generate new advertisements, or to determine when to present an advertisements.

[0163] In some cases, the observations may include textual or spoken instructions provided to the agent by a third-party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide textual or spoken instructions to the agent (e.g., to navigate to a particular location).

[0164] As another example, the environment may be an electrical, mechanical or electromechanical design environment, e.g., an environment in which the design of an electrical, mechanical or electro-mechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design the entity. The observations may comprise observations that characterize the entity, i.e., observations of a mechanical shape or of an electrical, mechanical, or electromechanical configuration of the entity, or observations of parameters or properties of the entity. The actions may comprise actions that modify the entity, e.g., that modify one or more of the observations. The design process may include outputting the design for manufacture, e.g., in the form of computer executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus a design of an entity may be optimized, e.g., by evolving a task-prompt that enables the task model to better design the entity, and then the optimized design output for manufacturing the entity, e.g., as computer executable instructions; an entity with the optimized design may then be manufactured.

[0165] As previously described the environment may be a simulated environment. Generally in the case of a simulated environment the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions. For example the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle. Generally the agent may be implemented as one or more computers interacting with the simulated environment.

[0166] As described above, any of the models described herein may be multimodal models. A few examples of machine learning tasks that may be performed by multimodal models follow.

[0167] For some tasks the second modality input represents an image or video as previously described, e.g. from a camera or other imaging device that captures the image or video from areal-world environment, and / or audio, e.g. audio data such as speech or other sounds captured from a real-world environment. In general the tasks described below may be tasks that require spatial awareness or other context from the image, video, or audio item. For example, a problem description may ask “What is the object in the top left comer?”, or “What was the answer to the spoken question?”.

[0168] As one example the task may comprise an object or action detection task. A taskspecific training example may comprise an image, video, or audio item containing one or more objects or actions, and a sequence of text. The sequence of text may describe or otherwise label the object(s) or action(s) and (for an image or video) may include text giving bounding box coordinates for the object(s) or action(s). A task-prompt evolved using the methods described herein may improve facilitate generation of a model output comprising or representing text that describes or otherwise labels detected object(s) or action(s) in the second modality input, and may (for an image or video) include bounding-box coordinates for the detected object(s) or action(s), e.g. " 10 20 90 100 cat 20 30 100 100 dog”.

[0169] As another example the task may comprise a classification task, e.g. an object or action classification task. A task-specific training example may comprise an image, video, or audio item containing one or more objects or actions and a sequence of text. The sequence of text may describe or otherwise classify the object(s) or action(s). A task-prompt evolved using the methods described herein may facilitate generation of a model output comprising data, e.g. text, that classifies the object(s) or action(s) in the second modality input into one of a plurality of classes.

[0170] As another example, the task may comprise an image, video, or audio item describing task, e.g. a captioning task (which, as used here, includes an audio description task to explain what is happening in a video). A task-specific training example may comprise an image, video, or audio item and a sequence of text describing the image, video, or audio item. A task-prompt evolved using the techniques described herein may facilitate generation of a model output comprising data, e.g. text, describing an image, video, or audio item in the second modality input. For example the model output may provide a caption or description for a second modality input item, or it may count objects in the second modality input item, or it may provide some other form of description of the second modality input item.

[0171] As another example the task may comprise an image, video, or audio questionanswering task. A task-specific training example may comprise an image, video, or audio item and a sequence of text that describes the image, video, or audio item. A task-prompt evolvedusing the methods described herein may facilitate generation of a model output comprising data, e.g. text, that answers a question about the second modality input specified in a prompt sequence of text, e.g. as described above. This may be used, e.g., to answer questions about visual plots and charts or about sounds.

[0172] As another example the task may comprise a character or word recognition task, e.g. an OCR (optical character recognition) task. A task-specific training example may comprise an image, video, or audio item and a sequence of text that includes text that is depicted in the image or video, or that is represented as speech in the audio item. A task-prompt evolved using the methods described herein may facilitate generation of a model output comprising text that represents characters or words in the second modality input, e.g. in a natural language.

[0173] As another example the task may comprise a still or moving image or audio generation task. A task-specific training example may comprise an image, video, or audio item and a sequence of text that describes the image, video, or audio item. A task-prompt evolved using the methods described herein may facilitate generation of a model output comprising data for an image, video, or audio item, e.g. image data defining values for pixels of a still or moving image or audio data representing values of an audio waveform, and the sequence of text in the multimodal input to the model may describe or characterize the image, video, or audio item to be generated.

[0174] As another example the task may comprise a computer language text generation task. A task-specific training example may comprise an image, video, or audio item and a sequence of text in a computer language for generating the image, video, or audio item. A task-prompt evolved using the methods described herein may facilitate generation of a model output comprising text in the or another computer language for generating or rendering an image, video, or audio item in the second modality input, e.g. a web page, plot, or chart.

[0175] In another example of a computer language text generation task a task-specific training example may comprise an image, video, or audio item and a sequence of text in a computer language for performing a task in relation to the image, video, or audio item, e.g. a data processing task that involves analyzing the content of the image, video, or audio item to provide a result of the analysis or, e.g., a search to search for information relating to the content of the image, video, or audio item. The computer language in the model output may comprise computer language for invoking a function or calling one or more external APIs. Merely as one example, such an output may be formatted as a JSON object. As previously, the sequence of text in the multimodal input may define the task to be performed and the second modalityinput may comprise, e.g. an image, video, or audio item in relation to which the task is to be performed, e.g. a task that involves manipulation of particular types of data that may benefit from access to an API such as mathematical data, date / time related data, scientific data, recent data that may post-date training of the model (that may be accessed by a search function or API), and so forth. A task-prompt evolved using the methods described herein may facilitate generation of a model output comprising text in the or another computer language for performing a task, e.g. as described above, in relation to an image, video, or audio item in the second modality input. The method may then include using the text in the computer language to perform the task.

[0176] In general where a model output comprises text this may be provided as speech representing the text.

[0177] In some implementations the task comprises an agent control task in which the agent interacts with an environment to perform the agent control task. In these implementations a multimodal input may include an observation characterizing the environment. For example the multimodal input can include a sequence of text that defines the task to be performed by the agent and the second modality input can represent an image, video, audio, or other observation of the environment, e.g. captured by a camera or other imaging device, or by a microphone, from a real-world environment. A task-specific training example may comprise a sequence of text representing one or more actions of the agent, and a second modality input representing an observation of the environment. A task-prompt evolved using the methods described herein may facilitate generation of a model output comprising an action selection output, e.g. including text, that is used to select one or more actions to be performed by the agent in the environment in response to the observation. As an illustration the task model output may define an action as text such as “A: 132 114 128 5 25 156”, that can be converted into a control signal for a mechanical agent, such as a robot, e.g. “AT = [0.1, —0.2,0] A / ? = [10°, 25°, —7°]”. As another example the action selection output may also or instead define one or more low-level skills, e.g. from a vocabulary of previously learnt skills. As before, the sequence of text in the multimodal input to the model may describe the task to be performed, e.g. “What action should the robot take to [perform task]”.

[0178] As another example, the environment may be a real-world computing environment and the task is to manage the processing, e.g. by one or more real-world servers, of a queue of continuously arriving jobs. The observations may comprise observations of the times of departures of successive jobs, or the time intervals between the departures of successive jobs,or the time a server takes to process each job, or the arrival times, or time intervals between the arrivals, of successive jobs, or data characterizing the type of job(s). The actions may comprise actions that allocate particular jobs to particular computing resources.

[0179] As another example, the environment may comprise a real-world computer system or network and the task may be to maintain security of the computer system or network. The observations may comprise any observations characterizing operation of the computer system or network, and the actions may comprise actions to control the operation e.g. to limit or correct abnormal or undesired operation e.g. because of the presence of a virus or other security breach.

[0180] As another example, the environment may comprise a data packet communications network environment, and the task may be to route packets of data over the communications network. The actions may comprise data packet routing actions and the observations may comprise, e.g., observations of a routing table which includes routing metrics such as a metric of routing path length, bandwidth, load, hop count, path cost, delay, maximum transmission unit (MTU), and reliability.

[0181] In some agent control implementations the agent may be a human agent and the environment may be a real -world environment. For example the agent can be a human user of a digital assistant such as a smart speaker, smart display, or some other device that is used to instruct the user to perform actions. The task may be any real-world task that the user wishes to perform. The observations may be obtained from an observation capture subsystem, e.g. a monitoring system such as a video camera or sound capture system, to capture visual and / or audio observations of the user performing the task. The actions may comprise instructions in the form of, e.g., text, image, video, or audio data such as speech, that guide the user in performing the task.

[0182] In each of the example tasks above, it will be appreciated that the evolution of the taskprompt may be specific to the domain of that task. In particular, the training tasks may comprise examples that are specific to the domain of that task. Alternatively, the evolution of the taskprompt may be non-domain-specific, that is general. In this case the training examples may include examples from a number of different task domains. In this way, a task-prompt may be generated that improves model inputs across a wide variety of task domains.

[0183] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out thoseoperations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0184] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0185] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed.Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0186] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0187] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0188] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0189] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The essential elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0190] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs),and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

[0191] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0192] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0193] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will dependon factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0194] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0195] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0196] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0197] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A computer-implemented method for generating a task-prompt for inclusion in a neural network input for controlling a neural network to perform a task, the method comprising: performing an evolutionary process using first and second processes, the evolutionary process comprising a plurality of evolutionary rounds performed on a population, the first process, comprising: providing a task-prompt mutation input as input to a task-prompt mutation neural network to generate a new task-prompt, the task-prompt mutation input comprising a mutation-prompt comprising an instruction to generate a new taskprompt based on at least one individual of the population; the second process, comprising: providing a mutation-prompt mutation input as input to a mutation-prompt mutation neural network to generate a new mutation-prompt, the mutation-prompt mutation input comprising a mutation-prompt and a hypermutation-prompt comprising an instruction to modify the mutation-prompt; wherein performing the evolutionary process comprises: for at least some of the evolutionary rounds: performing the first process to generate a new task-prompt to replace a taskprompt in the population; for at least some of the evolutionary rounds: performing the second process to generate a new mutation-prompt for use in performing the first process; evaluating a fitness of task-prompts of at least some individuals of the population; and selecting a task-prompt from the population based on respective finesses of the taskprompts.

2. The method claim 1, wherein for at least some of the evolutionary rounds, performing the first process comprises: selecting an individual from the population; and including in the task-prompt mutation input the task-prompt of the selected individual to generate an evolved task-prompt.

3. The method of claim 2, wherein selecting an individual from the population comprises: comparing two or more individuals from the population and selecting the individual comprising a task-prompt with the highest fitness of the selected individuals.

4. The method of claim 3, further comprising: replacing in the population at least one of the two or more individuals comprising a task-prompt with a lower fitness with a new individual comprising the evolved task-prompt.

5. The method of any preceding claim, wherein at least some individuals in the population further comprise a mutation-prompt.

6. The method of claim 5, wherein for at least some of the evolutionary rounds, performing the second process comprises: selecting an individual from the population; and including in the mutation-prompt mutation input the mutation-prompt of the selected individual to generate an evolved mutation-prompt.

7. The method of claim 6, wherein selecting an individual from the population comprises: comparing two or more individuals from the population and selecting the individual comprising a task-prompt with the highest fitness of the selected individuals.

8. The method of claim 7, further comprising: replacing in the population the individual comprising a task-prompt with a lower fitness with a new individual comprising the evolved mutation-prompt.

9. The method of claim 8 as dependent on claim 4, wherein the new individual comprises the evolved task-prompt and the evolved mutation-prompt.

10. The method of any preceding claim, further comprising initializing the population of individuals prior to performing the evolutionary process, comprising: selecting an initial mutation-prompt from a set of mutation-prompts;including the initial mutation-prompt in a prompt initialization input to a prompt initialization neural network to generate an initial task-prompt; and adding an individual comprising the initial task-prompt to the population.

11. The method of claim 10, wherein initializing the population comprises: selecting a modifier from a set of modifiers; and including the modifier in the prompt initialization input to the prompt initialization neural network to generate the initial task-prompt.

12. The method of claim 10 or 11, wherein initializing the population comprises: including a task specification in the prompt initialization input to the prompt initialization neural network to generate the initial task-prompt.

13. The method of any preceding claim, further comprising: maintaining, for at least some of the rounds of evolution, a set of one or more individuals having a highest fitness in that round of evolution; and for at least some of the evolutionary rounds: performing the first process comprises including the task-prompts of the set of one or more individuals in the task-prompt mutation input.

14. The method of claim 13, wherein the task-prompts of the set of one or more individuals is included in the task-prompt mutation input in chronological order.

15. The method of any preceding claim, wherein for at least some of the evolutionary rounds: performing the first process comprises: including a list of task-prompts in the task-prompt mutation input; and including in the task-prompt mutation input a mutation-prompt configured to cause the task-prompt mutation neural network to continue the list with one or more task-prompts.

16. The method of claim 15, further comprising generating the list of task-prompts, comprising:filtering the individuals of the population to remove individuals having a task-prompt that is within a threshold similarity of a task-prompt of another individual of the population.

17. The method of claim 15 or 16, wherein the list of task-prompts is included in the taskprompt mutation input in a random order.

18. The method of claim 15 or 16, wherein the list of task-prompts is included in the taskprompt mutation input in order of fitness.

19. The method of claim 18, further comprising including an order indication in the taskprompt mutation input indicating an order of the list of task-prompts.

20. The method of any preceding claim, wherein for at least some of the evolutionary rounds when performing the first process: providing the task-prompt mutation input comprising at least one individual comprises including in the task-prompt mutation input at least one reasoning path of the individual that resulted in a correct answer on at least one training example; and providing the task-prompt mutation input comprising a mutation-prompt comprises including in the task-prompt mutation input a mutation-prompt configured to instruct the task-prompt mutation neural network to generate a task-prompt that led to the reasoning path.

21. The method of any preceding claim, further comprising, for at least one evolutionary round: determining whether to perform a crossover operation in which a task-prompt of one individual in the population is replaced with a task-prompt of another individual in the population.

22. The method of any preceding claim, further comprising: maintaining, during the evolutionary process, a context buffer by adding to the context buffer reasoning paths that led to correct answers during evaluation of task-prompts; andfor at least one evolutionary round, determining whether to perform a context shuffling operation in which a reasoning path in the context buffer is replaced with another reasoning path.

23. The method of claim 22, further comprising: for at least one evolutionary round, determining whether to replace the entire context buffer with other reasoning paths.

24. The method of any preceding claim, wherein performing the first process comprises selecting a mutation strategy from a plurality of mutation strategies and selecting a mutationprompt based on the selected mutation strategy.

25. The method of any preceding claim, wherein the evolutionary process further uses a third process, the third comprising: providing a task-prompt generation input to a task-prompt generation neural network, the task-prompt generation input comprising an initial task specification and a fixed general prompt; and performing the evolutionary process comprises, for at least some of the evolutionary rounds, performing the third process to generate new task-prompts in the population.

26. The method of claim 25, wherein the fixed general prompt is configured to cause the task-prompt generation neural network to output a plurality of task-prompts and the third process further comprises selecting a task-prompt from the plurality of task-prompts.

27. The method of any preceding claim, wherein the evolutionary process further uses a fourth process, the fourth process comprising: providing a mutation-prompt generation input to a mutation-prompt generation neural network, the mutation-prompt generation input comprising an initial task specification and a thinking style.

28. The method of any preceding claim, wherein evaluating the fitness of a particular task-prompt on a training task comprises:providing a training input to a task neural network to generate an intermediate output, the training input comprising a task specification indicated in the training task and a first task-prompt; providing a second training input to the task neural network to generate a proposed solution to the task specification indicated in the training task, the second training input comprising the first training input, the intermediate output and the particular task-prompt; evaluating the solution with reference to a known solution of the training task.

29. The method of claim 28, further comprising, in at least one evolutionary round: generating the first task-prompt using the first process in the evolutionary round; and generating the particular task-prompt based on the first task-prompt using the first process in the same evolutionary round.

30. A computer-implemented method for generating a task-prompt for inclusion in a neural network input for controlling a neural network to perform a task, the method comprising: receiving a task specification; determining, based on the task specification, an initial task-prompt; providing a mutation-prompt mutation input as input to a mutation-prompt mutation neural network to generate a mutation-prompt, the mutation-prompt mutation input comprising an initial mutation-prompt and a hypermutation-prompt comprising an instruction to modify the mutation-prompt; and providing the mutation-prompt and the initial task-prompt to a task-prompt mutation neural network in a task-prompt mutation input to generate a new task-prompt.

31. The method of claim 30, comprising: evaluating a performance of the new task-prompt; and determining, based on the performance, to control the neural network using the new task-prompt.

32. The method of claim 31, wherein evaluating the performance of the new task-prompt comprises:providing a training input to a task neural network to generate an intermediate output, the training input comprising a task specification indicated in the training task and a first task-prompt; providing a second training input to the task neural network to generate a proposed solution to the task specification indicated in the training task, the second training input comprising the first training input, the intermediate output and the new task-prompt; evaluating the solution with reference to a known solution of the training task.

33. The method of any of claims 30 to 32, wherein determining the initial task-prompt comprises including the task specification in a prompt initialization input to a prompt initialization neural network to generate the initial task-prompt.

34. The method of claim 33, further comprising: including the initial mutation-prompt in the prompt-initialization input.

35. The method of claim 33 or 34, further comprising: including a modifier in the prompt initialization input.

36. The method of any of claims 30 to 35, wherein the initial task-prompt is one of a plurality of task-prompts provided in the task-prompt mutation input.

37. The method of claim 36, wherein the plurality of task-prompts comprises a list of task-prompts; and the mutation-prompt is configured to cause the task-prompt mutation neural network to continue the list with one or more task-prompts.

38. The method of claim 36 or 37, wherein the task-prompts of the plurality of taskprompts are included in the task-prompt mutation input in chronological order.

39. The method of claim 36 or 37, wherein the plurality of task-prompts is included in the task-prompt mutation input in a random order.

40. The method of claim 36 to 37, wherein the plurality of task-prompts is included in the task-prompt mutation input in order of performance.

41. The method of any of claims 36 to 40, further comprising including an order indication in the task-prompt mutation input indicating an order of the plurality of taskprompts.

42. The method of any of claims 30 to 41, further comprising: including in the task-prompt mutation input at least one reasoning path and an instruction for the task-prompt mutation neural network to generate a task-prompt that led to the reasoning path.

43. The method of any of claims 30 to 42, wherein the initial task-prompt mutation input comprises at least one example reasoning path; and the mutation-prompt is configured to instruct the task-prompt mutation neural network to generate a task-prompt that led to the at least one example reasoning path.

44. The method of claim 43, further comprising generating the at least one example reasoning path, the generating comprising: mutating an initial at least one example reasoning path based on a reasoning path mutation-prompt.

45. The method of any of claims 30 to 44, further comprising: maintaining a context buffer comprising one or more reasoning paths for inclusion in the task-prompt mutation input; and determining whether to perform a context shuffling operation in which a reasoning path in the context buffer is replaced with another reasoning path.

46. The method of claim 45, further comprising: determining whether to replace the entire context buffer with other reasoning paths.

47. A computer-implemented method for performing a task, comprising: receiving a task specification;determining, based on the task specification, an initial task-prompt; receiving an initial mutation-prompt; generating a mutation-prompt comprising mutating the initial mutation-prompt based on a hypermutation-prompt, wherein the hypermutation-prompt comprises an instruction to modify the mutation-prompt; and generating a new task-prompt comprising mutating the initial task-prompt based on the mutation-prompt.

48. The method of claim 47, comprising performing the task by controlling a neural network using the new task-prompt.

49. The method of claim 48, comprising: evaluating a performance of the new task-prompt; and determining, based on the performance, to control the neural network using the new task-prompt.

50. The method of claim 49, wherein evaluating the performance of the new task-prompt comprises: generating a training input comprising a task specification indicated in the training task and a first task-prompt; generating an intermediate output based on the training input; generating a second training input comprising the first training input, the intermediate output and the new task-prompt; generating a solution the task specification based on the second training input; and evaluating the solution with reference to a known solution of the training task.

51. The method of any of claims 47 to 50, wherein determining the initial task-prompt comprises generating the initial task-prompt based on the task specification.

52. The method of claim 51, wherein generating the initial task-prompt is further based on the initial mutation-prompt.

53. The method of claim 51 or 52, wherein generating the initial task-prompt is further based on a prompt modifier.

54. The method of any of claims 47 to 53, wherein the initial task-prompt comprises at least one example reasoning path; and the mutation-prompt is configured to instruct the task-prompt mutation neural network to generate a task-prompt that led to the at least one example reasoning path.

55. The method of claim 54, further comprising generating the at least one example reasoning path, the generating comprising: mutating an initial at least one reasoning path based on a reasoning path mutationprompt.

56. The method of any of claims 47 to 55, further comprising: wherein generating the new task-prompt is further based on at least one reasoning path in a context buffer; and the method further comprises determining whether to perform a context shuffling operation in which a reasoning path in the context buffer is replaced with another reasoning path.

57. The method of claim 56, further comprising: determining whether to replace the entire context buffer with other reasoning paths.

58. The method of any preceding claim, further comprising: receiving a user input comprising a problem description; modifying the user input to include the selected task-prompt; and providing the modified user input as input to a task neural network to generate a task neural network output.

59. The method of claim 58, wherein the user input indicates a security issue in a computing device, computer-network or in computer software code; the task neural network output comprises computer program code for execution by the device or computer network.

60. The method of claim 58, wherein the user input comprises input data derived from one or more sensors and indicating one or more properties of one or more physical objects in a real-word environment, the user input further comprising a request to output one or more instructions to complete a task including an action on or using the one or more physical objects; and the task neural network output comprises one or more instructions for execution by a real-world agent interacting with the environment.

61. The method of claim 60, further comprising controlling the real -world agent to perform action using the task neural network output.

62. The method of claim 58, wherein the user input comprises input data derived from one or more sensors and indicating one or more properties of a real-world environment, the user input further comprising a request to identify one or more conditions of the real-world environment based on the input data; and the task neural network output comprises an indication of whether the one or more conditions are present in the real-world environment.

63. The method of claim 58, wherein the user input indicates a portion of computer program code and a request to identify the presence of a bug in the computer program code; and the task neural network output comprises an indication of a bug in the computer program code.

64. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any preceding claim.

65. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one ormore computers, cause the one or more computers to perform operations of the respective method of any one of claims 1 to 63.

66. The system of claim 65, wherein the one or more computers comprise a first processor and a second processor configured to operate in parallel, and wherein: the first processor implements the task-prompt mutation model and the second processor implements the mutation-prompt mutation model in parallel; and / or the first processor performs the first process and the second processor performs the second process in parallel.

67. The system of claim 66, wherein the mutation-prompt mutation model has more parameters than the task-prompt mutation model.

68. The system of claim 65, wherein the one or more computers comprise a first one or more processors and a second processor; and wherein: the first one or more processors implement the task-prompt mutation model and the mutation-prompt mutation model and the second processor implements, in parallel, a task model configured to evaluate the performance of one or more task-prompts of at least some individuals of the population; and / or the first one or more processors perform the first and / or second processes and the second processor evaluates the performance of one or more task-prompts of at least some individuals of the population in parallel to the first one or more processors performing the first and / or second process.

69. The system of claim 66, wherein the first one or more processors comprises parallel hardware configured to implement the task-prompt mutation model and the mutation-prompt mutation model in parallel and / or to perform the first and second process in parallel.

Images