Method for generating a merged machine learning model

Abstract

A method for generating a merged machine learning language model from a plurality of pre-trained expert ML models uses a training dataset comprising data for reducing undesirable outputs from the merg

Description

Field

[001] The present techniques generally relate to a method for generating a merged machine learning, ML, model that is formed of multiple expert ML models that are each expert in performing a specific task. In particular, the present techniques provide a method for generating a merged ML model that is able to generate outputs that are both expert in multiple domains (i.e. for multiple tasks) and safe (i.e. do not contain undesirable content). Background

[002] Large Language Models (LLMs) have demonstrated impressive capabilities, often surpassing human performance across language processing tasks. To enhance performance in various domains, pre-trained LLMs are often finetuned on domain-specific data. Some examples of domain-specific expert models include OpenBioLLM, excelling in the biomedical domain, and MAmmoTH, performing well in STEM subjects. I.e. these models have been trained on expert knowledge in a specific area. Finetuning a model may also mean that the model has been trained to perform a specific task, for example, image segmentation or any other suitable task, and being able to perform a specific task may also be referred to as expert knowledge.

[003] However, such finetuned expert models perform well in their specific domains only and thus, there is a need to find models that are capable of performing well in several domains while retaining specific expert knowledge. Additionally, when merging multiple expert LLM models, while it is possible to create multi-domain expert models, there often is a transfer of misalignment to the merged model. That is, the merged model often has issues with safety, for example, the merged model may answer inappropriate questions, or may give inappropriate answers. When the task is, for example, an image segmentation task, merged models may be misaligned in how and what they segment. For example, image segmentation tasks in which a pedestrian is segmented from an otherwise busy street, a misaligned and / or unsafe model may no longer recognise pedestrians.

[004] The present applicant has identified the need for an improved technique to generate a machine learning, ML, model formed of a plurality of task specific expert models. Summary

[005] In a first approach of the present techniques, there is provided a computer-implemented method for generating a merged machine learning, ML, model from a plurality of pre-trained expert ML models, the method comprising: obtaining a plurality of expert ML models, wherein each expert ML model is pre-trained to perform a specific task, wherein each expert ML model comprises a set of model parameters for performing the specific task; obtaining a training dataset for determining how to merge the plurality of expert ML models, wherein the training dataset comprises data for reducing undesirable outputs from the merged model; and generating a merged ML model by: selecting one expert ML model to be a base model into which the remaining expert ML models are to be merged; determining, for each remaining expert ML model, a difference between the model parameters of the base model and the model parameters of the expert ML model, the difference corresponding to the specific task performed by the expert ML model; optimising, for each remaining expert ML model, a weighting factor using the determined difference and the training dataset, to be used when combining the set of model parameters of the expert ML model with the set of model parameters of the base model; and combining the set of model parameters of the base model with the set of model parameters of the expert models using the optimised weighting factors, to thereby generate a merged ML model for performing all of the specific tasks and reducing undesirable outputs.

[006] Advantageously, merging expert machine learning models that are each trained to perform a specific task means that the resulting merged model is capable of performing each of these tasks accurately and in a way that is desirable. Desirable may, for example, mean that no hallucinations, unethical or unsafe content is present in the output of the merged model. That is, outputs of the merged model are safe outputs, in that no untrue, derogatory or otherwise dangerous outputs are created. For example, the merged model may refuse to answer certain questions about topics which are deemed unsafe, such as weapons related topics or other inappropriate topics, such as information about poisons or toxins. This ensures that the merged ML model cannot be used to generate harmful output. At the same time, being able to merge two expert / composite models ensures accurate output with minimal training. In this way, merged models can be tailored to perform specific tasks with minimal training resources needed. Instead of completely retraining a model to perform well in more than one area, it is possible to create such a model from pre-trained expert models, without comprising accuracy or security of the output generated.

[007] Obtaining the plurality of expert ML models may comprise obtaining a plurality of expert Large Language Models, LLMs. In this case, generating the merged ML model may comprise generating a merged Large Language Model, LLM. LLMs are designed for natural language processing tasks such as language generation. That is, the merged model may be a model that is able to process and respond to a variety of user text-based queries (or speech-based queries that are converted to text-based queries). In particular, the expert LLMs referred to herein may be LLMs that are each trained to respond to queries in one particular area of expertise, such as science, maths, history or any other area. For example, one expert LLM may be trained to answer biology questions, while another expert LLM may be trained to answer maths questions. Their merged LLM may then be capable of answering both biology and maths questions. Similarly, the expert LLMs may each be expert at providing particular types of text-based responses to queries, such as: summarization, translation, smart reply (e.g. to help compose emails or text messages), tone adjustment (to alter the tone of text), Question &Answering, etc.

[008] Obtaining the plurality of expert ML models may comprise obtaining a plurality of expert diffusion models. In this case, generating the merged ML model may comprise generating a merged diffusion model. A diffusion model is a type of generative model. A diffusion model has three main components: a forward process, a reverse process, and a sampling procedure. The goal of a diffusion model is to learn a diffusion process for a given dataset, so that the process can generate new data that are similar to the data in the dataset. Diffusion processes are stochastic process and used to model real-life stochastic systems. Generally speaking, diffusion models take an input and gradually add noise to the input over a series of (time) steps. Then the diffusion model is trained to recover the original input from the noisy version, also over a series of (time) steps. Diffusion models may be used when the overall merged ML model is for image generation or audio generation.

[009] In some cases, the expert ML models may be in the form of adapters, or may comprise adapters. An adapter model is a smaller ML model that can be combined with a larger pretrained machine learning model. For example, a larger ML model may be capable of answering user questions in a larger number of areas. However, the larger ML model may not have expert level knowledge in all areas. For example, knowledge in biology may be missing, or the larger ML model may not always answer all biology related questions correctly. In another example, the larger ML model may be able to translate English to Korean, but not English to Japanese. An adapter that is trained to translate English to Japanese could be added into the larger ML model to provide the larger ML model with the missing functionality. This is advantageous because multiple large ML models are not needed for the same type of task, which would make ML models difficult to implement on constrained-resource devices such as smartphones. Using an adapter model means that only the adapter model needs to be trained on training data for a specific area in which the larger ML model should become an expert. This has the advantage that the knowledge retained in the larger ML model is still being used, but it is not necessary to completely retrain the larger ML model which is very computationally expensive. Instead, only the adapter model needs to be trained to achieve good results in the area in which the ML model is an expert. This further has the advantage that when two or more expert models are merged, only their adapter models need to be merged, if both are using the same larger ML model as their backbone. This is again computationally more efficient, as less training is required.

[010] In such cases, obtaining the plurality of expert ML models may comprise: obtaining a plurality of adapter models, each adapter model being pre-trained to perform a specific task. Then, generating the merged ML model may comprise merging the adapter models. (The larger, backbone, ML model into which the adapter models are inserted may be kept frozen during the merging, for the reasons mentioned above).

[011] As noted above, the expert ML models are merged with the base ML model by optimising the weighting factors used when combining all the model parameters. Combining all the model parameters of the expert ML models in this way may lead to higher accuracy for the individual tasks.

[012] Optimising the weighting factors may comprise using a set of weighting factors that balance the performance of the merged ML model on the different tasks, i.e. the weighting factors balance the importance of individual expert models. There are multiple ways in which to determine the weighting factors - two of these are described below in turn.

[013] One way to determine the weighting factors may comprise selecting, using an evolutionary-based algorithm, the set of weighting factors. Evolutionary-based algorithms may reproduce essential elements of the biological evolution in a computer algorithm in order to solve “difficult” problems, at least approximately, for which no exact or satisfactory solution methods are known. Evolutionary algorithms often perform well approximating solutions to all types of problems because they ideally do not make any assumption about the underlying fitness landscape. That is, such an algorithm may be an efficient and accurate way of finding a set of merged weights. The specific process to use the evolutionary-based algorithm is described below with reference to the Figures.

[014] In the case where an evolutionary-based algorithm is used, optimising weighting factors may comprise: evaluating a loss associated with the merged ML model when the merged ML model processes the training dataset, wherein the loss comprises a first loss corresponding to the specific tasks and a second loss corresponding to reducing undesirable outputs; and optimising the set of weighting factors to minimise the loss. In other words, the training dataset is not used to train the merged ML model, but is used to determine how to optimise the weighting factors used to generate the merged ML model. The training dataset is input into the merged ML model and the outputs are used to evaluate the loss.

[015] Another way to determine the weighting factors may comprise: identifying, using a subset of the obtained training dataset, the set of weighting factors based on a performance of the merged model on the subset of the training dataset. That is, the weighting factors are determined based on a few samples from the training dataset, where the samples include ground truths that can be used to analyse the performance. A grid search algorithm may be used, which is a hyperparameter optimisation technique that uses exhaustive searching through a manually specified subset of the hyperparameter space of a learning algorithm. For each point, performance is determined and grid search allows for the hyperparameters which perform best to be found.

[016] In this case, optimising a weighting factor may comprise: evaluating a loss associated with the merged ML model, wherein the loss is between merged model predictions and ground truth answers; and optimising the set of weighting factors to minimise the loss. In other words, the training dataset is not used to train the merged ML model, but is used to determine how to optimise the weighting factors used to generate the merged ML model. The training dataset is input into the merged ML model and the outputs are used to evaluate the loss.

[017] The step of obtaining a training dataset for the optimisation process may comprise: obtaining a first set of training data for task-specific evaluation of the merged ML model (i.e. for optimising the weighting factors for the specific tasks performed by the expert ML models); and obtaining a second set of training data for optimising the weighting factors to reduce undesirable outputs. That is, off-the-shelf training data may be utilised for both the first set and second set.

[018] Additionally or alternatively, obtaining a training dataset may comprise: generating the first set of training data and / or generating the second set of training data. That is, custom training data may be generated for the first set and / or second set. This may be useful when no off-the-shelf training data exists. As explained below, ML models may be used for the generating. Using ML models to generate training data has the advantage that a large number of training datapoints / data items can be created easily and quickly. When training data is generated and / or labelled manually, the process is very slow and required a lot of manual input, naturally limiting the number of training datapoints available.

[019] Generating the first set of training data for optimising the weighting factors for the specific tasks may comprise: generating, using each expert ML model, a plurality of task-specific questions and a plurality of task-specific answers that are specific to the task of the expert ML model; and combining the task-specific questions and task-specific answers from each expert ML model to form the first set of training data. That is, domain-specific training data may be generated, for each expert model, such that the generated training data is tailored towards the expertise of each of the expert ML models that are being merged. The expert models that are being merged may themselves be used to generate the first set of training data, i.e. the unmerged expert models may each be used to generate training data which is subsequently be used to determine the weighting factors that decide the make-up of the merged model. This is an efficient approach as the merged model should still be able to retain the knowledge and expertise of each of the expert models, and therefore, checking this using training data generated by the expert models themselves is an easy way to verify that the merged model still produces accurate results.

[020] Generating the second set of training data for optimising the weighting factors to reduce undesirable outputs may comprise: generating, using an uncensored LLM, a set of unsafe questions, wherein unsafe questions are those that are likely to generate an undesirable output from the merged ML model; inputting the set of unsafe questions into each expert ML model to generate a set of answers; and identifying, using an LLM, each refusal answer, wherein a refusal answer is an answer generated by an expert ML model when a real answer is considered to be unsafe; and combining the set of unsafe questions and corresponding refusal answers to form the second set of training data. Accuracy and safety of an output of an Al model are not always related. Therefore, it can be difficult to train a ML model to produce both accurate and safe / desirable output. By identifying and highlighting those outputs that are desirable, i.e. safe, the merged ML model can be trained to provide both accurate and safe outputs. This is important because the merging process may otherwise result in a misaligned model, which may have acquired domain knowledge but no longer produces desirable / safe results. Again, using ML models to generate this training data means that a large amount of data can be generated very quickly to train the merged model. This also means that it becomes feasible to quickly generate new merged models, with different combinations of experts depending on the task.

[021] Combining the set of unsafe questions and corresponding refusal answers may comprise: discarding any unsafe question for which no refusal answer is generated by any of the expert ML models. In other words, if all the expert ML models generate an answer for the unsafe question, it can be assumed that the unsafe question is not actually unsafe (i.e. can be answered in a safe way). It is the questions that any one expert ML model refuses to give an answer for that are more useful for training the merged ML model to reduce undesirable outputs.

[022] The specific tasks of the expert ML models may be specific text generation tasks. For example, the expert models may support translation of English into Korean, translation of English into Japanese, and translation of English into Spanish. Similarly, the expert models may parse unstructured data, and generate spreadsheets. Other, non-limiting example text generation tasks that could be provided by expert ML models include: text summarization, translation, smart reply (for automatically generating appropriate responses in email replies or text messages), tone adjustment (to alter the tone of text to suit a particular purpose, e.g. more formal for a job application, to more jovial for an after-dinner speech), Question &Answering (i.e. providing answers to user-specific questions that are received as text or speech), and so on.

[023] The specific tasks may be specific image generation tasks, and the training data may comprise text prompts and corresponding generated images that have been generated using the text prompts.

[024] The first set of training data may comprise text prompts and corresponding generated images that are examples of accurate performance for the image generation tasks; and the second set of training data may comprise text prompts and corresponding generated images that are examples of undesirable performance for the image generation tasks. For example, undesirable generated images may be images that comprise harmful content, such as hate, violence etc. Such content may, for example, be generated by a specific malicious model which is trained to output malicious content.

[025] The specific tasks may be audio generation tasks, and the training data may comprise audio prompts and corresponding generated audio samples that have been generated using the audio prompts. This may be used to generate speech or music, for example.

[026] The first set of training data may comprise audio prompts and corresponding generated audio samples that are examples of accurate performance for the audio processing tasks; and the second set of training data may comprise audio prompts and corresponding generated audio samples that are examples of undesirable content in generated audio. For example, undesirable content may include harmful words that relate to, for example, hateful or violent content. Similarly, such content may be generated by a malicious model that is designed to generate malicious content for training purposes.

[027] In a second approach of the present techniques, there is provided an apparatus for generating a merged machine learning, ML, model from a plurality of pre-trained expert ML models, the apparatus comprising: at least one processor coupled to memory for: obtaining a plurality of expert ML models, wherein each expert ML model is pre-trained to perform a specific task, wherein each expert ML model comprises a set of model parameters for performing the specific task; obtaining a training dataset for determining how to merge the plurality of expert ML models, wherein the training dataset comprises data for reducing undesirable outputs from the merged model; and generating a merged ML model by: selecting one expert ML model to be a base model into which the remaining expert ML models are to be merged; determining, for each remaining expert ML model, a difference between the model parameters of the base model and the model parameters of the expert ML model, the difference corresponding to the specific task performed by the expert ML model; optimising, for each remaining expert ML model, a weighting factor using the determined difference and the training dataset, to be used when combining the set of model parameters of the expert ML model with the set of model parameters of the base model; and combining the set of model parameters of the base model with the set of model parameters of the expert models using the optimised weighting factors, to thereby generate a merged ML model for performing all of the specific tasks and reducing undesirable outputs.

[028] The features described above with respect to the first approach apply equally to the second approach and therefore, for the sake of conciseness, are not repeated.

[029] As noted above, the apparatus comprises at least one processor and memory. The memory may store instructions that, when executed by the at least one processor individually or collectively, cause the at least one processor to perform the above-described steps.

[030] The apparatus may be a server, such that the merged ML model is generated on a server before being shared with and deployed on multiple user devices.

[031] The apparatus may be a user device such as a smart device, such that the merged ML model is generated on-device. The user device may be a smartphone. A smartphone is an example of a smart device. The user device may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV), a smart fridge, a smart oven, a smart vacuum cleaner, a smart robotic device, a smart lawn mower, and so on. More generally, the user device may be a constrained-resource device, but which has the minimum hardware capabilities to train a merged ML model as described above. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart lawn mower, smart oven, etc). It will be understood that this is a non-exhaustive and nonlimiting list of example devices.

[032] In a third approach of the present techniques, there is provided a computer-implemented method for generating a training dataset for use when generating a merged machine learning, ML, model formed of a plurality of expert ML models, the method comprising: generating a first set of training data for optimising weighting factors to be used to produce a merged ML model that performs specific tasks of all the expert ML models; and generating a second set of training data for optimising the weighting factors to be used to produce a merged ML model that reduces undesirable outputs.

[033] Generating the first set of training data for task-specific optimisation may comprise: generating, using each expert ML model, a plurality of task-specific questions and a plurality of task-specific answers that are specific to the task of the expert ML model; and combining the task-specific questions and task-specific answers from each expert ML model to form the first set of training data.

[034] Generating the second set of training data for reducing undesirable outputs from the merged ML model may comprise: generating, using an uncensored LLM, a set of unsafe questions, wherein unsafe questions are those that are likely to generate an undesirable output from the merged ML model; inputting the set of unsafe questions into each expert ML model to generate a set of answers; and identifying, using an LLM, each refusal answer, wherein a refusal answer is an answer generated by an expert ML model when a real answer is considered to be unsafe; and combining the set of unsafe questions and corresponding refusal answers to form the second set of training data.

[035] Combining the set of unsafe questions and corresponding refusal answers may comprise: discarding any unsafe question for which no refusal answer is generated by any of the expert ML models.

[036] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by at least one processor, causes the processor to carry out any of the methods described herein.

[037] In the cases where the present techniques are implemented or executed on a device comprising multiple processors, the present techniques may be implemented by one or more of the multiple processors. That is, the present techniques may be implemented by or executed by the processors individually or collectively.

[038] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[039] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[040] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[041] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[042] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[043] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[044] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[045] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[046] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[047] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[048] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[049] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[050] Figure 1 is a schematic diagram showing how the present techniques may be used compared to prior art techniques;

[051] Figure 2A is a flowchart showing example steps to train a merged ML model;

[052] Figure 2B is a flowchart showing example steps to generate training data for training a merged ML model;

[053] Figure 3 is a schematic diagram showing how synthetic data is being generated;

[054] Figure 4 is a table comparing performance of different merged models for alignment, i.e. safety, and accuracy;

[055] Figure 5 shows that the present safety-aware merging achieves the highest alignment across all scenarios;

[056] Figures 6A and 6B show varying loss combination factor a;

[057] Figure 7A shows results for generated data vs. real data;

[058] Figure 7B shows the importance of K, i.e., the number of samples in the first dataset ©expert and the second dataset ©safety;

[059] Figure 7C shows how accuracy and alignment change with the number of optimisation steps;

[060] Figure 8 shows an example prompt used to generate a first set of training data, i.e. domain data;

[061] Figure 9 shows an example prompt used to generate a first set of training data, i.e. alignment / safety data;

[062] Figure 10 shows alignment results for a merged models comprising seven expert models; and

[063] Figure 11 is a block diagram for an apparatus for implementing the present techniques. Detailed description of the drawings

[064] Broadly speaking, the present techniques generally relate to a method for generating a merged machine learning, ML, model from a plurality of pre-trained expert ML models. Advantageously, the present techniques enable a merged ML model to be created from a plurality of expert models, while maintaining safety and accuracy. That is, the merged ML model maintains alignment and is able to accurately answer expert questions that relate to a specific task, while not outputting any undesirable or unsafe answers, such as answers to unsafe questions that could be used to do harm.

[065] Merging Large Language Models (LLMs) is a cost-effective technique for combining multiple expert LLMs into a single versatile model, retaining the expertise of the original ones. However, current approaches often overlook the importance of safety alignment during merging, leading to highly misaligned models. The present techniques involve investigating the effects of model merging on alignment. Several popular model merging techniques are evaluated, and it is demonstrated that existing methods do not only transfer domain expertise but also propagate misalignment. The present techniques provide a simple two-step approach to address this problem: (i) generating synthetic safety and domain-specific data, and (ii) incorporating these generated data into the optimization process of existing data-aware model merging techniques. This allows alignment to be treated as a skill that can be maximized in the resulting merged LLM. Experiments illustrate the effectiveness of integrating alignment-related data during merging, resulting in models that excel in both domain expertise and alignment.

[066] Since expert models may excel in specific domains only, model merging has been proposed as a technique to combine the strengths of various models into a single, highly capable one. For instance, merging a model proficient in chemistry with another model expert in mathematics aims to create a unified model that performs well in both subjects, often outperforming the individual experts. This approach is particularly attractive as it allows leveraging the knowledge from numerous open-source models without incurring high training costs. However, a crucial question is posed that has been overlooked in the literature: how does model merging impact the safety alignment of existing LLMs?

[067] Figure 1 is a schematic diagram showing how the present techniques may be used compared to prior art techniques. To understand the importance alignment, a few notions about safety alignment are first introduced. Safety alignment refers to a model’s ability to generate responses that are safe, ethical, and consistent with human values. Safety alignment may also refer to a model’s ability to generate desirable output and maybe be referred to as generating desirable output herein. Herein, a model is referred to as “aligned” if the model has a high safety alignment and / or is highly likely to generate desirable output. Conversely, the model is “misaligned” if it is lacking necessary safety alignment or routinely generates undesirable output, as one of the expert models 114 in Figure 1. The present Applicant has found that naively merging 112 a set of expert LLMs including a misaligned model can result in a misaligned merged model 114, even if some of the original experts 110, 116 are aligned (Figure 1, left). This raises substantial concerns for the safe deployment of merged LLMs, which may expose users to unsafe and / or undesirable content. Hence, the need for safety-aware model merging 118 is shown, where merged models preserve desirable alignment characteristics 120 (Figure 1, right).

[068] To address this issue, the present techniques provide a simple yet effective approach to combine expert models while preserving alignment. The intuition behind the present techniques is that safety alignment / generation of desirable output should be considered as a task on its own, similar to domain-specific expertise in fields such as biology or physics, and thus it should be optimized for during merging. The present approach consists of two stages. First, synthetic data is generated to use for merging. Then, building on existing techniques, the generated data is used to perform a data-driven merging optimization procedure, preserving both the alignment and the expertise of the original models.

[069] Figure 2A is a flowchart showing the steps of the method of the present techniques. The method is for generating a merged machine learning, ML, model from a plurality of pre-trained expert ML models. The method comprises: obtaining a plurality of expert ML models, wherein each expert ML model is pre-trained to perform a specific task, wherein each expert ML model comprises a set of model parameters for performing the specific task (step S100); obtaining a training dataset for determining how to merge the plurality of expert ML models, wherein the training dataset comprises data for reducing undesirable outputs from the merged model (step S102); and generating a merged ML model by: selecting one expert ML model to be a base model into which the remaining expert ML models are to be merged (step S104); determining, for each remaining expert ML model, a difference between the model parameters of the base model and the model parameters of the expert ML model, the difference corresponding to the specific task performed by the expert ML model (step S106); optimising, for each remaining expert ML model, a weighting factor using the determined difference and the training dataset, to be used when combining the set of model parameters of the expert ML model with the set of model parameters of the base model (step S108); and combining the set of model parameters of the base model with the set of model parameters of the expert models using the optimised weighting factors, to thereby generate a merged ML model for performing all of the specific tasks and reducing undesirable outputs (step S110).

[070] Step S100 of obtaining the plurality of expert ML models may comprise obtaining a plurality of expert Large Language Models, LLMs. In this case, generating the merged ML model may comprise generating a merged Large Language Model, LLM. LLMs are designed for natural language processing tasks such as language generation. That is, the merged model may be a model that is able to process and respond to a variety of user text-based queries (or speech-based queries that are converted to text-based queries). In particular, the expert LLMs referred to herein may be LLMs that are each trained to respond to queries in one particular area of expertise, such as science, maths, history or any other area. For example, one expert LLM may be trained to answer biology questions, while another expert LLM may be trained to answer maths questions. Their merged LLM may then be capable of answering both biology and maths questions. Similarly, the expert LLMs may each be expert at providing particular types of text-based responses to queries, such as: summarization, translation, smart reply (e.g. to help compose emails or text messages), tone adjustment (to alter the tone of text), Question &Answering, etc.

[071] Step S100 of obtaining the plurality of expert ML models may comprise obtaining a plurality of expert diffusion models. In this case, generating the merged ML model may comprise generating a merged diffusion model. Diffusion models may be used when the overall merged ML model is for image generation or audio generation.

[072] In some cases, the expert ML models may be in the form of adapters, or may comprise adapters. An adapter model is a smaller ML model that can be combined with a larger pretrained machine learning model. For example, a larger ML model may be capable of answering user questions in a larger number of areas. However, the larger ML model may not have expert level knowledge in all areas. For example, knowledge in biology may be missing, or the larger ML model may not always answer all biology related questions correctly. In another example, the larger ML model may be able to translate English to Korean, but not English to Japanese. An adapter that is trained to translate English to Japanese could be added into the larger ML model to provide the larger ML model with the missing functionality. This is advantageous because multiple large ML models are not needed for the same type of task, which would make ML models difficult to implement on constrained-resource devices such as smartphones. Using an adapter model means that only the adapter model needs to be trained on training data for a specific area in which the larger ML model should become an expert. This has the advantage that the knowledge retained in the larger ML model is still being used, but it is not necessary to completely retrain the larger ML model which is very computationally expensive. Instead, only the adapter model needs to be trained to achieve good results in the area in which the ML model is an expert. This further has the advantage that when two or more expert models are merged, only their adapter models need to be merged, if both are using the same larger ML model as their backbone. This is again computationally more efficient, as less training is required.

[073] In such cases, step S100 of obtaining the plurality of expert ML models may comprise: obtaining a plurality of adapter models, each adapter model being pre-trained to perform a specific task. Then, generating the merged ML model may comprise merging the adapter models. (The larger, backbone, ML model into which the adapter models are inserted may be kept frozen during the merging, for the reasons mentioned above).

[074] At step S102, training data is obtained. Training data can be obtained from a database. For example, training data can be training data for ensuring that the merged model generates an accurate and desirable output.

[075] That is, there may be training data to ensure an accurate output and training data to ensure a desirable output. Accurate means that the merged model outputs true statements and does not, for example, hallucinate an answer. For example, a merged model which has been trained in a subject area, such as biology, should only output answers to biology related questions that are true.

[076] Desirable may mean that the merged ML model does not generate output that is inappropriate or dangerous. For example, the merged ML model should refuse to answer questions such as “How do I kill a person”, or “How do I poison food”. Instead, the merged ML model should generate a safe output, such as for example, stating that it cannot help, or that you should not poison food.

[077] Thus, step S102 of obtaining a training dataset for the optimisation process may comprise: obtaining a first set of training data for task-specific evaluation of the merged ML model (i.e. for optimising the weighting factors for the specific tasks performed by the expert ML models); and obtaining a second set of training data for optimising the weighting factors to reduce undesirable outputs. That is, off-the-shelf training data may be utilised for both the first set and second set.

[078] Additionally or alternatively, obtaining a training dataset may comprise: generating the first set of training data and / or generating the second set of training data. That is, custom training data may be generated for the first set and / or second set. This may be useful when no off-the-shelf training data exists. As explained below, ML models may be used for the generating. Using ML models to generate training data has the advantage that a large number of training datapoints / data items can be created easily and quickly. When training data is generated and / or labelled manually, the process is very slow and required a lot of manual input, naturally limiting the number of training datapoints available.

[079] Generating the first set of training data for optimising the weighting factors for the specific tasks may comprise: generating, using each expert ML model, a plurality of task-specific questions and a plurality of task-specific answers that are specific to the task of the expert ML model; and combining the task-specific questions and task-specific answers from each expert ML model to form the first set of training data. That is, domain-specific training data may be generated, for each expert model, such that the generated training data is tailored towards the expertise of each of the expert ML models that are being merged. The expert models that are being merged may themselves be used to generate the first set of training data, i.e. the unmerged expert models may each be used to generate training data which is subsequently be used to determine the weighting factors that decide the make-up of the merged model. This is an efficient approach as the merged model should still be able to retain the knowledge and expertise of each of the expert models, and therefore, checking this using training data generated by the expert models themselves is an easy way to verify that the merged model still produces accurate results.

[080] Generating the second set of training data for optimising the weighting factors to reduce undesirable outputs may comprise: generating, using an uncensored LLM, a set of unsafe questions, wherein unsafe questions are those that are likely to generate an undesirable output from the merged ML model; inputting the set of unsafe questions into each expert ML model to generate a set of answers; and identifying, using an LLM, each refusal answer, wherein a refusal answer is an answer generated by an expert ML model when a real answer is considered to be unsafe; and combining the set of unsafe questions and corresponding refusal answers to form the second set of training data. Accuracy and safety of an output of an Al model are not always related. Therefore, it can be difficult to train a ML model to produce both accurate and safe / desirable output. By identifying and highlighting those outputs that are desirable, i.e. safe, the merged ML model can be trained to provide both accurate and safe outputs. This is important because the merging process may otherwise result in a misaligned model, which may have acquired domain knowledge but no longer produces desirable / safe results. Again, using ML models to generate this training data means that a large amount of data can be generated very quickly to train the merged model. This also means that it becomes feasible to quickly generate new merged models, with different combinations of experts depending on the task.

[081] Combining the set of unsafe questions and corresponding refusal answers may comprise: discarding any unsafe question for which no refusal answer is generated by any of the expert ML models. In other words, if all the expert ML models generate an answer for the unsafe question, it can be assumed that the unsafe question is not actually unsafe (i.e. can be answered in a safe way). It is the questions that any one expert ML model refuses to give an answer for that are more useful for training the merged ML model to reduce undesirable outputs.

[082] Generating the merged ML model comprises optimising the set of weighting factors to minimise a loss, so that the merged ML model is both domain-specific and safety-aware.

[083] Step S104 of selecting one expert ML model to be a base model into which the remaining expert ML models are to be merged comprises making an arbitrary choice about which expert ML model is to be used as the base model, as explained below with reference to the background of model merging.

[084] Step S106 of determining, for each remaining expert ML model, a difference between the model parameters of the base model and the model parameters of the expert ML model, may comprise defining a task vector, for each expert model, which represents the difference between the mode parameters. This is explained in more detail below with reference to Equation 1.

[085] Step S108 of optimising, for each remaining expert ML model, a weighting factor may comprise using any technique for the optimisation. Two examples are described below - EvoMM and LM-Cocktail. The process involves minimising a loss of the merged model, as described below in more detail (see also Equation 5).

[086] Step S110 of combining the set of model parameters of the base model with the set of model parameters of the expert models using the optimised weighting factors, is explained in more detail below.

[087] Figure 2B is a flowchart showing example steps to generate training dataset for use when generating a merged machine learning, ML, model formed of a plurality of expert ML models. The method comprises: generating a first set of training data for optimising weighting factors to be used to produce a merged ML model that performs specific tasks of all the expert ML models (step S200); and generating a second set of training data for optimising the weighting factors to be used to produce a merged ML model that reduces undesirable outputs (step S202).

[088] Step S200 of generating the first set of training data for task-specific optimisation may comprise: generating, using each expert ML model, a plurality of task-specific questions and a plurality of task-specific answers that are specific to the task of the expert ML model; and combining the task-specific questions and task-specific answers from each expert ML model to form the first set of training data.

[089] Step S202 of generating the second set of training data for reducing undesirable outputs from the merged ML model may comprise: generating, using an uncensored LLM, a set of unsafe questions, wherein unsafe questions are those that are likely to generate an undesirable output from the merged ML model; inputting the set of unsafe questions into each expert ML model to generate a set of answers; and identifying, using an LLM, each refusal answer, wherein a refusal answer is an answer generated by an expert ML model when a real answer is considered to be unsafe; and combining the set of unsafe questions and corresponding refusal answers to form the second set of training data.

[090] Combining the set of unsafe questions and corresponding refusal answers may comprise: discarding any unsafe question for which no refusal answer is generated by any of the expert ML models.

[091] Figure 3 is a schematic diagram showing how synthetic data is being generated. Synthetic data may be used to generate the merged ML model as described above with reference to Figure 2A. In more detail, to generate synthetic data, first two datasets of questions and associated answers are generated: one for preserving alignment, the other for transferring domain-specific knowledge. The first dataset contains “bad” or misaligned questions, that a malicious user may use to prompt an LLM. Both safety data ©safety (top) and expert domain data ©expert (bottom) are generated. For safety data, an uncensored LLM may be used to generate harmful questions, and refusals of the © experts may be collected with LLaMA-Guard. For domain data, the T experts may be used to generate questions in different domains (self-questioning) and collect responses.

[092] An example of such a prompt may be “How do I kill someone?”. Answers to these questions are then generated by the most aligned models in the pool of experts, typically taking the form of refusals (e.g., “I’m sorry, I can’t help.”). The second dataset contains domain-specific prompts, such as “What is the powerhouse of the cell?” for the biology domain. Domain-specific answers (e.g., “Mitochondria is the powerhouse of the cell.”) are provided by the most expert model in the pool on a specific domain. Finally, the collected data are used with data-driven merging approaches, such as those described in Xiao et al or Akiba et al, where minimizing a loss on both alignment and domain-specific data is optimised. By doing this, it is ensured that the merged model maintains high alignment and domain performance.

[093] The present techniques make at least the following contributions: a demonstration that existing model merging techniques fail to explore the inherent trade-off between alignment and domain accuracy; a new safety-aware merging pipeline that achieves greater alignment of the merged model without sacrificing its accuracy; and extensive experiments and ablations on the components of the present techniques, demonstrating their robustness under several conditions.

[094] Before explaining the present techniques in detail, some brief comments on existing techniques are provided to explain the challenges.

[095] LLM Alignment: Ensuring the alignment of LLMs is crucial. It has been shown that even benign datasets can degrade model safety and careful adaptation protocols are needed to preserve alignment. Recently, some techniques to align LLM were proposed. Some methods enhance generalization, while others propose separate alignment on helpfulness and harmlessness. Some techniques use synthetic data to reduce human supervision. Although these may be effective, it is shown herein that model merging can mitigate the effects of alignment procedures. Importantly, some existing techniques have addressed the need for effective input-output safeguarding in conversational Al with Llama Guard, employing a safety risk taxonomy and ad hoc models to classify safety concerns in text. r0961 Model Merging: Techniques for merging multiple models have been proposed as efficient ways to benefit from the capabilities of multiple LLMs without retraining or accessing the original datasets. One technique proposes to combine models with weight averaging, showing improved performance compared to a single model. Another builds on this by performing task arithmetic, i.e. element-wise operations on model parameters to edit their behaviour towards specific tasks. Model merging in non-linear spaces has shown improved results. Some techniques propose methods to improve model merging, focusing on sparsification. Similarly, some techniques exploits sparse masks for better combination. Importantly, some extend merging capabilities across multiple modalities. The importance of each model to merge can be automatically tuned with data-driven approaches such as EvoMM by Akiba et al and LM-Cocktail by Xiao et al. None of these approaches consider the safety implications of merging.

[097] Alignment Evaluation: Advancements in evaluating LLMs have focused on their robustness, ethical considerations, and safety alignment. However, the present Applicant is the first to evaluate the alignment of merged models.

[098] Background on Model Merging

[099] Consider an ensemble of N models T. Each f e T is a model that excels in a specific domain, outperforming other models in domain-specific benchmarks. Let us define onefbase e T as the base model, parameterized by 0base e ]Rd. The choice of the base model is arbitrary. Similarly, the remaining N - 1 expert models are defined as {fexpert}^-!1, each parameterized by ^expert G Kd.

[100] Following Iharco et al, a task vector xt e IRd is defined as the difference between the parameters of the expert and base models by ^expert ®base- (^) The set of task vectors is identified as Using task arithmetic, a merged model fmerged parameterized by 0merged g IRd can be obtained, transferring the knowledge of multiple experts while preserving the expertise of the base model. This is generally written as: ^merged ®base + St=l (2) where At e HR are task weighting factors that balance the performance on different tasks. Several approaches implement more advanced strategies for task vector combination, such as SLERP, TIES, DARE, or DARE-TIES. However, these still require manual tuning of the task weighting values At, to balance the importance of each model during merging.

[101] Automatic Task Weighting

[102] The choice of At values significantly influences the effectiveness of existing merging techniques. To address this issue, several methods for automatic selection of task weighting factors have been proposed. For instance, EvoMM is an evolutionary-based algorithm for selecting the At using an iterative genetic algorithm. In each iteration, {^}^1 values are randomly sampled p times, where p is a population hyperparameter typical of genetic optimization. Assuming a merging algorithm like TIES, this generates p different versions of ^merged, which are then evaluated according to a user-defined criterion C, such as accuracy on a downstream question-answering task evaluated on a set of datasets, for general or domain-specific knowledge evaluation. The goal of EvoMM is to find 0merged to maximize the performance, according to the criterion C. The genetic algorithm assesses the effectiveness over the entire population of sampled fmerged on In the next iteration, a new set of {^}^1 are sampled close to the At resulting in the best-performing fmerged- This process is repeated until convergence.

[103] Alternatively, LM-Cocktail proposes a method for identifying At based on performance on a few samples. Assuming a dataset 2) composed of a few domain-specific questions and answers (q, a), they design a heuristic that balances the contributions of existing models based on their performance on 2). This is formulated as: wt -^Ce(feXpert(Q)' a)L ,3' = softmaxCfwa^1), where £ce refers to the cross-entropy loss between the model prediction and the ground-truth answer. In LM-Cocktail, {Af}^1 are the terms of a linear combination of weights rather than of task vectors A common aspect of both approaches for automatic task weighting is the usage of external data. The present techniques exploit this characteristic to enforce safety alignment in merged models while maximizing accuracy.

[104] Motivation

[105] Recall that although merging techniques are effective for boosting performance on downstream datasets, an important aspect has been overlooked in the literature: there is no consideration of safety alignment in the merging process. Naively merging models with existing techniques can result in the removal of safety alignment, as shown later. This issue may prevent the deployment of merged models, where safety is required. The present techniques build on state-of-the-art data-dependent automatic task weighting strategies to propose simple baselines for safety-aware merging.

[106] The intuition behind the present techniques is that safety alignment should be treated as a task in its own right. Just as domain expertise is optimized, safety alignment must also be optimized during model merging. Current automatic task weighting methods rely on data to optimize performance and to achieve the present goal, it is necessary to incorporate both alignment data and domain data into the optimization process. By leveraging this data dependency, the present techniques ensure that the merged model retains both domain expertise and safety alignment incorporated in the data. Moreover, the present techniques provide a fully automated pipeline, relying on synthetic data only, while compatibility with public datasets is retained, which allows external dependencies in the merging process to be avoided.

[107] Safety Data Generation

[108] The goal of safety alignment in LLMs is to respond to unsafe input prompts with refusals, i.e., sentences like “I am sorry, but I cannot help”. When undesirable outputs are being avoided, an unsafe or undesirable output may also mean an output that is outside certain parameters and this does not necessarily require an unsafe input prompt. For example, an undesirable output when an LLM is used for image segmentation may also mean the LLM not segmenting out, for example, pedestrians. This can be both unsafe and undesirable when the segmented output is used to, for example, directa self-driving car. Safety alignment is typically achieved through fine-tuning on unsafe prompts and their corresponding refusals. However, models in the merging set T may have been trained with different data and procedures, leading to varying levels of safety alignment. Therefore, it is important that the merged model fmerged reproduces the refusals of models f e T for unsafe inputs.

[109] The present techniques start by generating a set of K unsafe questions ^safety An uncensored LLM may be used to generate ^safety, since safety-aligned LLMs in T may refuse to generate such questions. Details of the prompt are provided below. This can be replaced with pre-generated unsafe inputs from datasets such as BeaverTails. Then qsafety ~ ^safety is used as input for all f e T, collecting a set of replies for each prompt qsafety These replies are processed with LLaMA-Guard 2 to identify refusals. One refusal asafety is randomly selected for each qsafety. By repeating this for all qsafety 6 2Safety> a set °f refusals c / Zsafety is obtained. This results in a safety dataset of unsafe questions and associated refusals, ©safety = {(Safety Safety)}f=i> where q‘afety e Safety ^safety e ^safety The process is shown in Figure 3 (top). If no model in T replies with a refusal, the input qsafety is discarded.

[110] Domain Data Generation

[111] Besides preserving alignment, the present techniques aim to transfer the expertise of each fexpertto ^merged- To do this, a Q&A dataset is generated for each domain of expertise to optimize task weighting.

[112] The present techniques may use the expert models to generate questions. Each feXpert is prompted to generate an expert-specific question Expert- For instance, if feXpert specializes in mathematics, it may be used to generate math-related questions. In-context learning may be used to provide examples of questions. Then, feXpert may be prompted with Expertto obtain a corresponding answer ciexpert- This self-questioning procedure is inspired by related literature. Each model feXpert produces K / (N - 1) questions and associated answers, hence all questions and answers can be aggregated in two sets Expert and ^expert, respectively, both of size K. Finally, ©expert = {(Expert, Expert)}i=i is constructed, where q‘xpert e Sexpert-Expert e ^expert- This process is shown in Figure 3 (bottom). Existing datasets can also be used as an alternative, though this may require additional data collection or reliance on external sources that might be limited or not accessible for particular domains.

[113] Merging

[114] The present techniques use the previously collected datasets, ©safety and ©expert, to guide the optimization of task weights At, maximizing both alignment and domain performance. By leveraging automatic task weighting strategies that depend on data, such as EvoMM and LM-Cocktail, the present techniques ensure that the merged model retains both safety alignment and domain expertise. The present techniques propose a custom safety-aware adaptation of both EvoMM and LM-Cocktail.

[115] For EvoMM, the present techniques optimize the merged model fmerged to output an associated response a, given a question q, where the pair (q,a) is sampled from either ©safety or ©expert- This ensures that the resulting fmerged preserves both the safety alignment of existing models in © and their expertise in various domains. Formally, given (qsafety< “safety) ~ ©safety and (^expert, “expert) ~ ©expert, the present techniques impose a cross-entropy loss£ce between the answer generated by fme^edC?) and the associated reply a. The cross-entropy loss is applied to the logits for each predicted token. This is formulated as: ©r ^(qr,ar)~'Dr[ ©ce (fmerged (Qr)< “r)]< ,,, r e {safety, expert}. The two terms are combined into a single loss, using a factor a to balance each contribution by: ©merge ©safety T “©expert- (5)

[116] It is then assumed that C = ©merge and optimisation is over In other words, the present techniques use the merged model fmerged to process both ©expert and ©safety, optimizing {^}^1 to maximize performance on both. Recall that 0merged is obtained with Eq. (2).

[117] For LM-Cocktail, instead, the present techniques assume © = ©safety u ©expert. and calculate {^}^1 applying Eq. (3) for all {f^ert}"©!1 ■

[118] Figure 4 is a table comparing performance of different merged models for alignment, i.e. safety, and accuracy. That is, Figure 4 shows a benchmark of safety-aware merging. Performance is reported in in two different© setups, achieving aligned models expert in STEM and biology. Comparison with baselines is made by performing manual hyperparameter search (grid search) or using automatic task weighting strategies with ©expert only. The present safety-aware alignment techniques not only preserve better the highest safety alignment of merged models but also improve accuracy. Comparative gain is shown within brackets with respect to the baseline for each block.

[119] Experiments

[120] Experimental Setup

[121] Merging Techniques: The present techniques use two automatic methods to find the task weights of Eq. (2), i.e., EvoMM and LM-Cocktail, in which safety alignment data is added as described above. As recommended, the present techniques use EvoMM for optimization on top of DARE-TIES, and TIES and SLERP are added as merging algorithm for completeness. For all, the merged models maximizing domain accuracy are reported. The experiments use MergeKit as codebase.

[122] Models: Five LLMs are used for the experiments, i.e. Mistral-0.2-7B-lnstruct, LLaMA-3-8B-lnstruct, OpenBioLLM-8B, MAmmoTH-2-7B, and Wizard Math-1.1-7 B - in the following, versions are dropped for brevity. Among them, experts in the biology (OpenBioLLM), STEM (MAmmoTH), and math (WizardMath) domains are considered, as well as instruction-finetuned models (Mistral, LLaMA). General-purpose models (Mistral, LLaMA) are setasfbase. Note that although these models lack domain expertise, they are finetuned on safety instructions for refusals generation; hence, they exhibit safety properties that are desirable to preserve. For each expert, domain data Expertis generated following the self-questioning procedure introduced above with custom prompts, capturing specific expertise. The prompts for data generation are reported below when discussing Figure 9.

[123] Evaluation: To evaluate alignment, the BeaverTails30K test is used including 1,733 unsafe prompts, for which aligned language models are expected to generate refusals. Responses for each prompt are generated with the obtained models and LLaMA-Guard-2 is used for flagging the answers as safe or unsafe. Finally, the percentage of safe outputs (i.e. refusals) is reported as an alignment metric. For domain performance, specific benchmarks related to domain expertise are used. A STEM set is considered, composed of some STEM subjects, from MMLLI as defined in Azerbayev et al.,2023; a BIO set, composed of MedMCQA, MedQA-USMLE-4-options, PubMedQA, and six biology-related subjects from MMLLI: College Biology, College Medicine, Anatomy, Pro Medicine, Medical Genetics, and Clinical KG. The commonsense reasoning WinoGrande is also used and the science-related reasoning ARC. For each benchmark, the model accuracy is calculated on multiple choice or binary classification tasks with LM Harness.

[124] Safety-Aware Merging Performance

[125] Benchmark: Figure 4 is a table comparing performance of different merged models for alignment, i.e. safety, and accuracy. In this Figure, results are presented across merging configurations with N = 2. The aim is to obtain merged models with good domain expertise and desirable safety alignment. First, T = {Mistral, MAmmoTH} is considered, to obtain an aligned STEM expert. Here, performance is evaluated on the STEM set. In a second set of experiments, T = {LLaMA, OpenBioLLM} is considered, to get an aligned biology expert. For the latter, the accuracy is evaluated on the BIO set. The average accuracy across all datasets is reported in the splits. First performance of the models in T is verified for both setups. In Figure 4, first row, it is shown that base models are most aligned, with 91.5 alignment for Mistral and 97.9 for LLaMA. Expert models report better performance in domain-specific tasks, such as 53.1 for MAmmoTH on STEM (vs 49.6 for Mistral) and 71.8 for OpenBioLLM on BIO (vs 68.9 for LLaMA), while they both lack safety alignment (64.8 and 48.3, respectively).

[126] Then, strong grid search baselines are proposed, by extensively optimizing manually task weights and hyperparameters for the Tl ES, DARE-TIES, and SLERP merging algorithms. These baselines do not use auxiliary data for the optimization of task weights, but they require considerable computation times due to the multiple configurations available. Then, results for the data-driven strategies EvoMM and LM-Cocktail using ©expert only are presented. EvoMM is used to optimize and hyperparameters of the task vector combination algorithm (i.e , TIES, DARE-TIES, and SLERP), as detailed below. For LM-Cocktail, is optimised only.

[127] The safety-aware merging performance is reported, by including ©safety in each data-driven merging strategy. For EvoMM, it is shown that including safety data achieves the highest alignment of merged models, reporting for instance 96.1 in DARE-TIES with EvoMM, only 1.8 below the original LLaMA (98.0), while EvoMM using only ©expert falls short at 80.1. Also, it is highlighted how the present techniques achieve great accuracy across all setups, always outperforming single experts in © and, in many scenarios, even outperforming corresponding safety-unaware baselines. Indeed, while the base EvoMM does not surpass the extensive grid search baseline, incorporating the present safety alignment data significantly enhances its performance. This may be caused by the usage of data beyond ©expert. that help regularize the optimization process, converging to better minima forfAt}^1. EvoMM (ours) achieves the highest alignment across all scenarios while maintaining competitive accuracy compared to grid search. The results are consistent using LM-Cocktail too, where alignment is improved in both scenarios (+1.9 and +1.5, respectively) while achieving on-par domain accuracy compared to the baseline LM-Cocktail with only Expert-

[128] Merging Beyond Two Models: The potential of safety-aware merging is investigated with a pool of experts T encompassing more than two models. In this setup, T is considered to be composed by: Mistral, MAmmoTH, and WizardMath. This setup is specifically designed since, although both MAmmoTH and WizardMath are finetuned on similar domains, they exhibit significant differences in performance on the Winogrande and ARC benchmarks, as empirically verified in Figure 5. Indeed, while MAmmoTH is an expert on ARC, WizardMath outperforms all on Winogrande. Mistral is an expert in alignment, reporting 91.6 on BeaverTails30K.

[129] Results following this setup are reported in Figure 4. Note that SLERP is not applicable since it is only usable when N = 2. Figure 5 shows that the present safety-aware merging achieves the highest alignment across all scenarios. Additionally, it attains the best domainspecific accuracy in 5 out of 6 cases. Compared to two-model merging, EvoMM shows significant improvements over LM-Cocktail, benefiting from its greater flexibility. That is, Figure 5 shows merging for three models. Benchmarks of three models and their merged counterparts are shown. With the addition of 2)safety, both alignment and domain accuracy is considerably increased on WinoGrande (WG) and ARC, for both EvoMM and LM-Cocktail.

[130] Ablation studies

[131] Ablation studies are presented here. In these experiments, the focus is on the LLaMa-OpenBioLLM merge with TIES and EvoMM as the automatic task weighting strategy.

[132] Impact of a: The term a was introduced to balance the importance of the two loss terms ^safety and ^expert in Emerge (Eq. (5)). Figures 6A and 6B show varying loss combination factor a. The safety-aware setup is tested with different values of a in Figures 6A and 6B. It is highlighted that for a <0.5, performance does not vary much, proving the robustness of the present approach. Interestingly, even with a = 0, equivalent to using 2)safety data only, performance remains competitive in accuracy. This shows that safety data may sometimes be sufficient to drive the merging procedure towards an acceptable combination of A value of a = 0.3 is chosen as the value maximizing the accuracy and used for the experiments, yielding 73.6 accuracy and 96.0 alignment. Higher a ( e.g , a = 1) leads to saturation of the accuracy (73.4), but at a great cost for alignment (88.9). For a <0.5, merging yields good results in both accuracy (Figure 6B) and alignment (Figure 6A). For greater a ( e.g , 1.0), alignment degrades significantly while accuracy does not improve.

[133] Data Source: Above, the process to generate ©safety and ©expert using models in © has been explained, hence avoiding to reliance on external data. Here, performance with real data is tested, constructing ©expert and ©safety by collecting samples from the validation set of existing benchmarks. A collection of K = 1000 prompts is made from the BIO validation set, and a collection of K = 1000 instances is made from BeaverTails30K training set. The above methods are then used to generate responses to the collected questions. Note that although existing datasets are used, none of these samples are used during evaluation. Figure 7A shows results for generated data vs. real data. Real data significantly benefits the baseline EvoMM, improving accuracy by (+0.6) and alignment by (+9.9). In contrast, the present safety-aware pipeline shows minimal gains (+0.2) in both accuracy and alignment with real data, demonstrating the effectiveness of the present synthetic data approach. When using real data, both methods achieve comparable accuracy, but the present safety-aware EvoMM maintains a substantially higher alignment (+6.5). [1341 Number of Samples: Safety-aware merging requires K samples in each ©expert and ©safety Figure 7B shows the importance of K, showing results for K e {200,500,1000}. It can be seen that accuracy is marginally impacted by increasing K, while alignment is more heavily influenced, achieving 96.0 alignment for K = 1000, where the second best value is 95.6 for K = 200. Choosing K = 1000 for all the experiments achieves the best trade-off between accuracy and alignment. That is, Figure 7B ablates the effect of K, i.e , the number of samples in ©expert and ©safety

[135] Optimization Steps for EvoMM: Figure 7C shows how accuracy and alignment change with the number of optimisation steps. Evolutionary optimization algorithms such as CMA-ES are iterative in nature. The impact of the iterations in relation to merging performance is investigated. In Figure 7C, the optimization steps in EvoMM are varied. It can be seen that more iterations benefit alignment transfer, while accuracy decreases. It is observed that accuracy decreases in favour of alignment by increasing the number of optimization steps. This behaviour is attributed to the greater difficulty of the alignment task, requiring more steps to be effectively transferred in fmerged- Due to the increased optimization times for more steps, the experiments are performed with 100 steps, guaranteeing the best trade-off between performance and optimization times.

[136] In conclusion, the effects of model merging in the context of safety alignment for LLMs has been shown. In the experiments, it has been demonstrated that existing techniques may cause merged models to lose alignment, preventing a safe deployment. A simple safety-aware method is proposed by the present techniques, which is combined with the existing EvoMM and LM-Cocktail strategies for data-dependent merging. By treating alignment as a task in its own right and incorporating alignment data into the merging process, the present safety-aware merging pipeline significantly improves alignment, without compromising domain accuracy.

[137] Domain Data Generation

[138] Figure 9 shows the prompt employed for domain-specific data generation ©expertSpecific prompts are used for the construction of ©expert while using expert-generated data.

[139] Figure 8 shows an example prompt used to generate domain data. Each expert model Expertis prompted to generate expert-specific questions and associated answers with the prompt shown in Figure 8. Here, “biology” is set as topic for BIO, “STEM” for STEM, and “reasoning” for ARC and WinoGrande. A total of 3 in-context samples are used, selected randomly from the ensemble of validation sets of all considered datasets for a specific domain. Note that these in-context samples are easy to obtain and serve as a guide for the generation process in the target domain. After generation, a post-processing step is performed where questions are deduplicated, and any presence of the used in-context prompt in the generated list of prompts is eliminated by exact match deduplication. Questions are generated in English.

[140] While associated misaligned answers are collected (as shown in the prompt), those are not used. Instead, reliance is only on refusals obtained by processing the unsafe questions with the models f e © as explained above.

[141] Safety Data Generation

[142] The construction of the safety data set©safety = {^safety- ^safety} is discussed. Again, as explained above, the present safety-aware merging is tested with real data sampled from the training set of BeaverTails30K.

[143] To generate 2)safety synthetically, Dolphin-2.9-LLama3-8b is prompted as outlined above. The prompt used to generate the misaligned requests is shown in Figure 10. A postprocessing deduplication is performed to ensure variability.

[144] The responses to those prompts are generated with the models in the pool T, obtaining refusals used for alignment as described above.

[145] Implementation Details

[146] Model settings: For generating responses, a greedy generation is employed, by setting the temperature of the sampling process in LLM inference to 0. This is done for both c / Zsafety and c / Zexpert. The models were allowed to generate up to 512 tokens. For faster processing, HuggingFace inference is used with distributed generation.

[147] Genetic optimization details

[148] The size of the initial population for genetic optimization was determined using the CMA-ES suggested formula: p = 4 + [3 • log(n)J where n is the number of parameters to optimize for. In the present case, n refers to the union of and specific hyperparameters for each merging strategy (see below). Each EvoMM merge was run on 4 A100 GPUs, taking approximately 45 minutes to complete. The total computational costs for the entire experiments described herein amounts to 50 A100 GPU days.

[149] Grid search details

[150] For the TIES and DARE-TIES models, combinations of two hyperparameters were considered: density dDT and weight wDT. The weight parameter refers to the interpolation factor, while the density parameter pertains to the sparsification of the task vectors. Reference is made to Yadav et al. and Yu et al. for details. All combinations were tested for dDT = {0.25,0.5,1.0} and wDT = {0.25,0.5,1.0} when two models are in the pool T ( i.e , N = 2). Considering the experiment with N = 3 (see above), instead, testing is with wDT = {0.1,0.25,0.33,0.5,1.0}, while dDT is unchanged. For SLERP, only the weight parameter wSL is ablated in the range wSL = {0.1,0.2,...,1.0}. In Figures 4 and 5, the result achieving the best domain accuracy is reported.

[151] Existing Model Merging Overlook Alignment

[152] The widespread habit among users of open-source models to merge models without safety considerations and upload them to public repositories like HuggingFace has been noticed. This poses a significant risk of proliferating highly misaligned models.

[153] Figure 10 shows alignment rates of a popular merged model on HuggingFace. To illustrate this issue concretely, the publicly available model martyn / mistral-megamerge-dare-7b (https: / / hqqqinqface.Go / m3rt¥n / mi^rahm^ is considered, which has been downloaded over 3,000 times by June 2024. This model was created using the DARE merging technique using 7 models available in HuggingFace, which is shown in Figure 10.

[154] Figure 10 shows alignment results for a merged models comprising seven expert models. The alignment rate of each model is calculated, using LLaMA-Guard 2 as described above. The alignment rates of the seven models in T vary between 61.8 and 93.0. However, the resulting merged model martyn / mistral-megamerge-dare-7b exhibits a poor alignment rate of 53.0, which is even less than the least aligned models in T, being them uukuguy / speechless-code-mistral-7b-v1.0 and lntel / neural-chat-7b-v3-3 exhibiting 61.8 alignment. The merged model (last row) achieves significantly lower alignment than all other models in T.

[155] This observation raises concerns about current merging practices and the subsequent deployment and uploads to public repositories. It is empirically observed that this model does not exhibit consistently better performance than the individual ones, and, as such, this was considered as a less interesting case study for the experiments.

[156] Figure 11 is a block diagram for an apparatus for implementing the present techniques. The apparatus 100 comprises at least one processor 202 coupled to memory 204 for implementing the present techniques. The apparatus also comprises a merged model 206 which is run by the at least one processor 202 coupled to memory 204. The merged model is generated by merging a plurality of expert models 208 the merged models using training data 210 and the methods described herein (e.g. with respect to Figure 2A).

[157] References: • LLaMA-Guard - Meta. 2024. Llama guard 2.https: / / github.com / meta- llama / PurpleLlama / blob / main / Llama-Guard2 / MODEL_CARD.md. • EvoMM I Akiba et al - Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2024. Evolutionary optimization of model merging recipes. arXiv:2403.13187. • LM-Cocktail I Xiao et al - Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. 2023. LM-cocktail: Resilient tuning of language models via model merging. arXiv:2311.13534. • Iharco et al - Gabriel llharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In ICLR. • SLERP - Tom White. 2016. Sampling generative networks. arXiv: 1609.04468. • TIES - Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. In NeurlPS. • DARE - Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML. • DARE-TIES - Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML. &Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s mergekit: A toolkit for merging large language models. arXiv:2403.13257. • Uncensored LLM - https: / / huggingface.co / cognitivecomputations / dolphin-2.9-llama3-8b • BeaverTails - Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2024. Beavertails: Towards improved safety alignment of llm via a humanpreference dataset. NeurlPS. • LLaMA-Guard 2 - Meta. 2024. Llama guard 2. https: / / github.com / meta-llama / PurpleLlama / blob / main / Llama-Guard2 / MODEL_CARD.md. • MergeKit - Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s mergekit: A toolkit for merging large language models. arXiv:2403.13257. • Mistral-0.2-7B - Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, etal. 2023. Mistral 7b. arXiv:2310.06825. • LLaMA-3-8B - AI@Meta. 2024. Ilama3 / MODEL CARD.md at main • meta-llama / llama3 • OpenBioLLM-8B - Malaikannan Sankarasubbu Ankit Pal. 2024. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https: / / huggingface.co / aaditya / OpenBioLLM-Llama3-70B. • MAmmoTH-2-7B - Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. Mammoth: Building math generalist models through hybrid instruction tuning. In ICLR. • WizardMath-1.1-7B - Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv:2308.09583 • MMLU - Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In ICLR. • Azerbayev et al - Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An open language model for mathematics. In ICLR. • MedMCQA - Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In CHIL. • MedQA-USMLE-4-options- Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences. • PubMedQA - Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In EMNLPIJCNLP. • College Biology, College Medicine, Anatomy, Pro Medicine, Medical Genetics, and Clinical KG - Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In ICLR. • WinoGrande - Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM. • ARC - Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 Reasoning Challenge. arXiv: 1803.05457. • LM Harness - Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.A framework for few-shot language model evaluation. • CMA-ES- Nikolaus Hansen, Sibylle D Muller, and Petros Koumoutsakos. 2003. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary Computation. • Dolphin-2.9-LLama3-8b - https: / / huggingface.co / cognitivecomputations / dolphin-2.9-Ilama3-8b • HuggingFace - https: / / huggingface.co • Yadav et al. - Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. In NeurlPS. • Yu et al. - Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML.

[158] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

1. A computer-implemented method for generating a merged machine learning, ML, model from a plurality of pre-trained expert ML models, the method comprising:obtaining a plurality of expert ML models, wherein each expert ML model is pre-trained to perform a specific task, wherein each expert ML model comprises a set of model parameters for performing the specific task;obtaining a training dataset for determining how to merge the plurality of expert ML models, wherein the training dataset comprises data for reducing undesirable outputs from the merged model; andgenerating a merged ML model by:selecting one expert ML model to be a base model into which the remaining expert ML models are to be merged;determining, for each remaining expert ML model, a difference between the model parameters of the base model and the model parameters of the expert ML model, the difference corresponding to the specific task performed by the expert ML model;optimising, for each remaining expert ML model, a weighting factor using the determined difference and the training dataset, to be used when combining the set of model parameters of the expert ML model with the set of model parameters of the base model; andcombining the set of model parameters of the base model with the set of model parameters of the expert models using the optimised weighting factors, to thereby generate a merged ML model for performing all of the specific tasks and reducing undesirable outputs.

2. The method as claimed in claim 1 wherein:obtaining the plurality of expert ML models comprises obtaining a plurality of expert Large Language Models, LLMs.

3. The method as claimed in claim 1 wherein:obtaining the plurality of expert ML models comprises obtaining a plurality of expert diffusion models.

4. The method as claimed in any of claims claim 1 to 3 wherein:obtaining the plurality of expert ML models comprises: obtaining a plurality of adapter models, each adapter model being pre-trained to perform a specific task.

5. The method as claimed in any preceding claim wherein optimising a weighting factor for each remaining expert ML model comprises:selecting, using an evolutionary-based algorithm, the set of weighting factors.

6. The method as claimed in claim 5 wherein optimising a weighting factor for each remaining expert ML model comprises:evaluating a loss associated with the merged ML model, wherein the loss comprises a first loss corresponding to the specific tasks and a second loss corresponding to reducing undesirable outputs; andoptimising the set of weighting factors to minimise the loss.

7. The method as claimed in claim 4 wherein optimising a weighting factor for each remaining expert ML model comprises:identifying, using a subset of the obtained training dataset, the set of weighting factors based on a performance of the merged model on the subset of the training dataset.

8. The method as claimed in claim 7 wherein optimising a weighting factor for each remaining expert ML model comprises:evaluating a loss associated with the merged ML model, wherein the loss is between merged model predictions and ground truth answers; andoptimising the set of weighting factors to minimise the loss.

9. The method as claimed in any preceding claim wherein obtaining a training dataset comprises:obtaining a first set of training data for optimising the weighting factors for the specific tasks; andobtaining a second set of training data for optimising the weighting factors to reduce undesirable outputs.

10. The method as claimed in claim 9 wherein obtaining a training dataset comprises:generating the first set of training data and / or generating the second set of training data.

11. The method as claimed in claim 10 wherein generating the first set of training data comprises:generating, using each expert ML model, a plurality of task-specific questions and a plurality of task-specific answers that are specific to the task of the expert ML model; andcombining the task-specific questions and task-specific answers from each expert ML model to form the first set of training data.

12. The method as claimed in claim 10 or 11 wherein generating the second set of training data comprises:generating, using an uncensored LLM, a set of unsafe questions, wherein unsafe questions are those that are likely to generate an undesirable output from the merged ML model;inputting the set of unsafe questions into each expert ML model to generate a set of answers; andidentifying, using an LLM, each refusal answer, wherein a refusal answer is an answer generated by an expert ML model when a real answer is considered to be unsafe; andcombining the set of unsafe questions and corresponding refusal answers to form the second set of training data.

13. The method as claimed in claim 12 wherein combining the set of unsafe questions and corresponding refusal answers comprises:discarding any unsafe question for which no refusal answer is generated by any of the expert ML models.

14. The method as claimed in any of claims 1 to 13 wherein the specific tasks of the expert ML models are specific text generation tasks.

15. The method as claimed in any of claims 1 to 13 wherein the specific tasks of the expert ML models are specific image generation tasks.

16. The method as claimed in any of claims 1 to 13 wherein the specific tasks of the expert ML models are specific audio generation tasks.

17. An apparatus for generating a merged machine learning, ML, model from a plurality of pre-trained expert ML models, the apparatus comprising:at least one processor coupled to memory for:obtaining a plurality of expert ML models, wherein each expert ML model is pretrained to perform a specific task, wherein each expert ML model comprises a set of model parameters for performing the specific task;obtaining a training dataset for determining how to merge the plurality of expert ML models, wherein the training dataset comprises data for reducing undesirable outputs from the merged model; andgenerating a merged ML model by:selecting one expert ML model to be a base model into which the remaining expert ML models are to be merged;determining, for each remaining expert ML model, a difference between the model parameters of the base model and the model parameters of the expert ML model, the difference corresponding to the specific task performed by the expert ML model;optimising, for each remaining expert ML model, a weighting factor using the determined difference and the training dataset, to be used when combining the set of model parameters of the expert ML model with the set of model parameters of the base model; andcombining the set of model parameters of the base model with the set of model parameters of the expert models using the optimised weighting factors, to thereby generate a merged ML model for performing all of the specific tasks and reducing undesirable outputs.

18. A computer-implemented method for generating a training dataset for use when generating a merged machine learning, ML, model formed of a plurality of expert ML models, the method comprising:generating a first set of training data for optimising weighting factors to be used to produce a merged ML model that performs specific tasks of all the expert ML models; and generating a second set of training data for optimising the weighting factors to be used to produce a merged ML model that reduces undesirable outputs.

19. The method as claimed in claim 18 wherein generating the first set of training data for task-specific training comprises:generating, using each expert ML model, a plurality of task-specific questions and a plurality of task-specific answers that are specific to the task of the expert ML model; andcombining the task-specific questions and task-specific answers from each expert ML model to form the first set of training data.

20. The method as claimed in claim 18 or 19 wherein generating the second set of training data for training the merged ML model to reduce undesirable outputs comprises:generating, using an uncensored LLM, a set of unsafe questions, wherein unsafe questions are those that are likely to generate an undesirable output from the merged ML model;inputting the set of unsafe questions into each expert ML model to generate a set of 5 answers; andidentifying, using an LLM, each refusal answer, wherein a refusal answer is an answer generated by an expert ML model when a real answer is considered to be unsafe; andcombining the set of unsafe questions and corresponding refusal answers to form the second set of training data.1021. The method as claimed in claim 20 wherein combining the set of unsafe questions and corresponding refusal answers comprises:discarding any unsafe question for which no refusal answer is generated by any of the expert ML models.1522. A computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out the method of any of claims 1 to 16, or 18 to 21.A

Images