Personalized image generation

Abstract

The present techniques relate to a machine learning method and apparatus for generating personalized images. In particular, the present techniques provide an image generation system which generates an

Description

Field

[001] The present techniques generally relate to a method and apparatus for generating personalized images. In particular, the present techniques provide an image generation system which generates an output image based on user selected content and style as well as a method for training the image generation system. Background

[002] With the introduction of text-to-image generation models based on denoising diffusion, the quality of the generated images has significantly improved. Furthermore, recently there has been growing interest in personalized image generation, where users can provide just a few images of reference subjects or styles to generate images that depict such subjects or styles in various contexts.

[003] A key enabler of this personalization breakthrough is Low-Rank Adaptation (LoRA), for example as described in “Low-rank adaptation of large language models” by Hu et al published in 2021. This paper describes a parameter-efficient method that achieves high-quality personalization using only a few training samples. This innovation has spurred extensive model sharing on open-source platforms like Civitai and Hugging Face. The immediate availability of pre-trained LoRAs has fueled interest amongst generative Al practitioners in trying to combine them to create images of personal subjects in various styles. For instance, a user might apply a concept (subject) LoRA trained on a few photos of their pet, and combine it with a downloaded style LoRA to render their pet in an artistic style of their choice.

[004] Simply averaging the LoRA weights can work acceptably when the subject and style share significant visual characteristics. However, in most cases, fine-tuning of the merging weights (i.e. the coefficients used to combine / merge LoRAs) is required. Merely as an example, “ZipLoRA: Any subject in any style by effectively merging loras” by Sohn et al published in European Conference on Computer Vision in 2024, describes an approach that directly optimizes merging coefficients through a customized objective function tailored to each subject-style LoRA combination. However, ZipLoRA’s reliance on optimization for each new combination incurs a substantial computational cost, typically taking minutes (e.g. 10 minutes) to complete. This limitation restricts its practicality for real-time applications on resource-constrained devices like smartphones.

[005] The present applicant has identified the need for an improved technique for generating personalized images which is useable on a resource-constrained device. Summary

[006] In a first approach of the present techniques, there is provided a computer-implemented method for real-time generation of a personalized image using a generative machine learning, ML, model, the method comprising: receiving, from a user, content input indicating content to be included in the generated image; receiving, from a user, style input indicating a style for the generated image; obtaining a content adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image containing the indicated content; obtaining a style adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image in the indicated style; predicting, using a hypernetwork, merging coefficients for merging the obtained content adapter and the obtained style adapter; merging, using the predicted merging coefficients, the obtained content adapter and the obtained style adapter together to form a merged adapter; combining the merged adapter with the generative ML model to adapt the generative ML model to generate an image containing the indicated content in the indicated style; and generating, using the combined generative ML model and merged adapter, the image comprising the indicated content in the indicated style.

[007] Advantageously, the present techniques include a hypernetwork to produce weight coefficients for merging subject and style related parameters. A hypernetwork is a network which generates the parameters of other networks (e.g. the adapters). The hypernetwork is typically separate from the generative ML model as detailed above and is used to form an enhanced generative ML model (i.e. generative ML model combined with merged adapter). This means the techniques allow generalization to unseen subject-style combinations which is not possible in real-time with known techniques. The present techniques can run in realtime (e.g. in around a second or less) on either a server or a user device such as a mobile device, despite the obtaining, predicting, merging and combining steps happening each time before generating the image. The generative ML model may be any suitable image generation model, e.g. a diffusion model. The generative ML model may be downloaded at inference time.

[008] There are many use case examples. For example, the generated image may be used as wallpaper or screensaver on a phone or other electronic device, e.g. a digital TV or fridge. The present techniques enable personalization and generation of personal content / subject (e.g. a personal pet) in various styles. Selected contents in existing images may be replaced by personalized content and / or styles. The generated image may be created using any suitable app e.g. via the gallery app or the portrait studio app.

[009] Receiving the user content input may comprise receiving an image of the content and receiving the user style input may comprise receiving an image which is in the indicated style. These images may be received directly from the user or may be received in response to a request to select the image(s). The images may be photos or may be sketches. For example, the user content input may comprise receiving a text input indicating both the content and the style for the image which is to be generated and the user may be requested to select at least one image of the indicated content and / or at least one image in the indicated content. The user may thus provide a text input such as “a close-up image of a dog in a Van Gogh style painting” and the user may be requested to select images of the dog (e.g. four samples). The user may also be requested to classify the indicated content / subject, e.g. “please enter a name for your dog”. Other examples of text input include “Hi Samsung TV! I’d like to see a screensaver of my dog Zoe wearing a hero costume in a professional photographer style”.

[010] Receiving the user content input may comprise receiving a content adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image containing the desired / indicated content. Similarly, receiving the user style input may comprise receiving a style adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image containing in the desired / indicated style. The user may select the content and / or style adapter from a library or database of such adapters. Similarly, when the content and / or style input are an image and / or text input, the content and / or style adapter may be obtained from a library or database of such adapters. The adapters in the database may have been pre-trained as described in more detail below.

[011] Each of the content and / or style adapters is a machine learning ML model which comprises a set of trainable parameters. Each adapter may be termed an adapter module and the terms may be used interchangeably. The set of parameters for each adapter may be significantly smaller than the set of parameters for the generative ML model (which may be termed a base model). The adapter may be combined with the generative ML model using any suitable technique, e.g. the adapter parameters may be applied to any subset of weight matrices in the generative ML model to reduce the number of trainable parameters. Even though the adapters are small and lightweight, the merged adapter does not need to be stored on device, just the individual adapters and the merging coefficients so that the merged adapter can be generated on-the-fly each time it is used. Moreover, any storage may be temporary for the generation of the image.

[012] Each adapter may be a Low-Rank Adaptation (LoRA) as described, for example in “Low-rank adaptation of large language models” by Hu et al. published in 2021 or “ZipLoRA: Any subject in any style by effectively merging loras” by Sohn et al. published in European Conference on Computer Vision in 2024. Hypernetworks are described for example in “HyperDreamBooth: Hypernetworks for Fast personalization of Text-to Image models” by Ruiz et al published in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2023. A model 2) that uses an adapter, e.g. a LoRA adapter L may be denoted as T)L = 2) © L with weights Wo + &W, where operation © means the adapter L is applied to the base model 2). To specify content and style, two separate adapters Lc (content adapter) and Ls (style adapter) with respective weight update matrices &WC and AM^ may be used.

[013] Calculating merging coefficients may comprise predicting merging coefficients in a column-wise fashion. In other words, by extracting a column in the weight update matrix for the content adapter; extracting the corresponding column in the weight update matrix for the style adapter; concatenating the extracted columns and repeating the extracting and concatenating steps for each column in the weight update matrices. Concatenating the extracted columns means joining the columns end-to-end, e.g. to form a single vector. The concatenated columns may then be input into the hypernetwork and the hypernetwork may output a plurality of merging coefficients which comprise a merging coefficient for each column of each content adapter and a merging coefficient for each column of each style adapter. These coefficients may then be used to merge the adapters Lc and Ls, resulting in the merged adapter Lm with update matrix AWm = mc 0 AWC + ms 0 A

[014] The hypernetwork may comprise a plurality of input layers. Inputting the concatenated columns into the hypernetwork may further comprise: combining the concatenated columns to form a concatenated matrix, selecting an input layer in the hypernetwork based on a size of the concatenated matrix (e.g. dimension of the matrix); and inputting the concatenated columns into the selected input layer of the hypernetwork. The number of input layers for the hypernetwork may be selected based on a counted number of unique layer shapes in the generative ML model. For example, for a diffusion-based model SDXL, there may be three input layers. There may be no hidden layers in the hypernetwork.

[015] Considered weight matrices may comprise some or all of query weights Wq, key weights Wk, value weights Wv and output weights Wo. LoRAs may be applied to the query, key, value and output attention component weights of the generative ML model for example as described in “A note on LoRA” by Fomenko et al published in 2024. Merging coefficients may be predicted by the hypernetwork for one or more of these different types of weights, e.g. both query and output weights, all of query, key and output weights, both query and value weights or just value weights. The other types of weights, for which no predicted merging coefficients have been generated may be merged using averaged merging coefficients (i.e. equally weighted coefficients of 0.5 and 0.5).

[016] The generative ML model may be in the form of a diffusion ML model described in High-Resolution Image Synthesis with Latent Diffusion Models by Rombach et al. that was presented in the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2022. The model may use a UNet architecture proposed by Ronneberger et al. in “U-Net: Convolutional Networks for Biomedical Image Segmentation” that was presented in the International Conference on Medical Image Computing and Computer-Assisted Intervention in 2015. A diffusion ML model is a type of generative model. A diffusion ML model has three main components: a forward process, a reverse process, and a sampling procedure. The goal of a diffusion ML model is to learn a diffusion process for a given dataset, so that the process can generate new data that are similar to the data in the dataset. Diffusion processes are stochastic process and used to model real-life stochastic systems.

[017] As set out above, the method may comprise receiving from a user, a text input indicating both the content and the style for the generated image. For such an input, the image may be generated using a generative ML model in the form of a text-to-image generative ML model which may be a diffusion ML model. Generating the image may further comprise: generating, using a text decoder, a text embedding of the text input; inputting, into the diffusion ML model, the generated text embedding together with noise as a latent seed to obtain an initial latent image representation; iteratively denoising the initial latent image representation to obtain an output latent image representation; and converting, using a decoder, the output latent image representation to the generated image. The generation of the text embedding may be considered to be the forward process and the conversion using the decoder may be considered to be the reverse process. The use of noise as a latent seed and the iterative denoising may be considered to form a sampling process.

[018] After generating the image, the method may further comprise assessing the generated image using a multimodal large language model, MLLM, e.g. “LLaVA-Critic: Learning to evaluate multimodal models” by Xiong et al. published in 2024. The assessment may comprise generating a first prompt for assessing whether the content in the generated image matches the indicated content and separately generating a second prompt for assessing whether the style of the generated image matches the indicated style. The method may then comprise inputting the first and second prompts and the generated image into a multimodal large language model, MLLM; and outputting from the MLLM an evaluation of whether the generated image accurately represents the indicated content in the indicated style. Outputting an evaluation from the MLLM may comprise: outputting, in response to the first prompt, a first assessment of whether the content in the generated image matches the indicated content; outputting, in response to the second prompt, a second assessment of whether the style of the generated image matches the indicated style; and outputting an evaluation that the generated image accurately represents the indicated content in the indicated style when the first assessment indicates that there is matching content and the second assessment indicates that there is a matching style. In other words, binary ratings may be used for both style and content evaluations. These binary ratings may then be combined with an image deemed correct only if it fulfills both evaluations.

[019] The hypernetwork may be trained prior to generating an image by: selecting, from a dataset of adapters, a pair of adapters wherein each pair comprises a content adapter and a style adapter; predicting, using the hypernetwork, merging coefficients for the selected pair of adapters; merging, using the predicted merging coefficients the selected pair of adapters to form a merged adapter; combining the merged adapter with the generative ML model; generating, using the combined generative ML model and merged adapter, an image comprising content corresponding to the selected content adapter in the style of the selected style adapter; and updating the parameters of the hypernetwork to minimize a merging loss for the generated image. It will be appreciated that the training may be a separate aspect of the present techniques.

[020] According to a second aspect of the present techniques, there is provided a computer-implemented method for training a hypernetwork to be used with a generative ML model and a pair of adapters for real-time generation of a personalized image, the training comprising: selecting, from a dataset of adapters, a pair of adapters wherein each pair comprises a content adapter and a style adapter, the content adapter is combinable with the generative ML model to adapt the generative ML model to generate an image containing content corresponding to the selected content adapter and the style adapter is combinable with the generative M L model to adapt the generative ML model to generate an image in a style corresponding to the selected content adapter; predicting, using the hypernetwork, merging coefficients for the selected pair of adapters; merging, using the predicted merging coefficients the selected pair of adapters to form a merged adapter; combining the merged adapter with the generative ML model; generating, using the combined generative ML model and merged adapter, an output image comprising content corresponding to the selected content adapter in the style of the selected style adapter; and updating the parameters of the hypernetwork to minimize a merging loss for the generated image.

[021] The following features may apply to both aspects.

[022] Each of the training steps may be repeated iteratively for a pre-determined number (e.g. k) steps. The training may comprise generating, using the combined generative ML model and content adapter, a first image; generating, using the combined generative ML model and style adapter, a second image; and calculating a merging loss from a sum of a difference between the generated output image and the first image and a difference between the generated output image and the second image. For example, the merging loss £merge may defined as: ■Emerge =H © ^m)(.xc>Pc^ — © Lc)(%c, pc) ll2 + || (2) ©Lm)(xs,ps) - (2) © LsXxs,ps) ||2+A|mc • ms| where xc, xs are the noisy latent image representations, pc, ps are the text prompts for content and style reference images respectively, the term A controls the strength of the orthogonalitypromoting regularization term, 2) is the base model, Lm is the merged adapter, Lc is the selected content adapter and Ls is the selected style adapter and ||.. ||2 is the L2 norm difference. Using this loss, the update to the parameters of the hypernetwork may be defined as: M M ~ IjVjfLmerge

[023] It will be appreciated that by adapting the parameters of the hypernetwork, the predicted merging coefficients will thus be changed for the next iteration and hence the merging loss reduced. The training may be carried out on a server.

[024] In a third approach of the present techniques, there is provided a user device comprising at least one processor and memory storing instructions that, when executed by the at least one processor individually or collectively, cause the user device to receive, from a user, content input indicating content to be included in the generated image; receive, from a user, style input indicating a style for the generated image; obtain a content adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image containing the indicated content; obtain a style adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image in the indicated style; calculate, using a hypernetwork, merging coefficients for merging the obtained content adapter and the obtained style adapter; merge the obtained content adapter and the obtained style adapter to form a merged adapter; combine the merged adapter with the generative ML model; and generate, using the combined generative ML model and merged adapter, the image comprising the indicated content in the indicated style.

[025] In a fourth approach of the present techniques, there is provided a server comprising at least one processor and memory storing instructions that, when executed by the at least one processor individually or collectively, cause the server to train a hypernetwork to be used with a generative M L model and a pair of adapters for real-time generation of a personalized image, by selecting, from a dataset of adapters, a pair of adapters wherein each pair comprises a content adapter and a style adapter, the content adapter is combinable with the generative ML model to adapt the generative ML model to generate an image containing content corresponding to the selected content adapter and the style adapter is combinable with the generative ML model to adapt the generative ML model to generate an image in a style corresponding to the selected content adapter; predicting, using the hypernetwork, merging coefficients for the selected pair of adapters; merging, using the predicted merging coefficients the selected pair of adapters to form a merged adapter; combining the merged adapter with the generative ML model; generating, using the combined generative ML model and merged adapter, an output image comprising content corresponding to the selected content adapter in the style of the selected style adapter; and updating the parameters of the pair of adapters to minimize a merging loss for the generated image.

[026] The features described above with respect to the second approach apply equally to the fourth approach and therefore, for the sake of conciseness, are not repeated.

[027] The user device may be a smart device. The user device may be a smartphone. A smartphone is an example of a smart device. The user device may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV), a smart fridge, a smart oven, a smart vacuum cleaner, a smart robotic device, a smart lawn mower, and so on. More generally, the user device may be a constrained-resource device, but which has the minimum hardware capabilities to use the personalized ML model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart lawn mower, smart oven, etc). It will be understood that this is a non-exhaustive and non-limiting list of example devices.

[028] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by at least one processor, causes the processor to carry out any of the methods described herein.

[029] In the cases where the present techniques are implemented or executed on a device comprising multiple processors, the present techniques may be implemented by one or more of the multiple processors. That is, the present techniques may be implemented by or executed by the processors individually or collectively.

[030] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[031] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[032] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object-oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[033] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[034] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general-purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[035] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[036] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[037] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[038] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[039] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[040] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[041] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[042] Figure 1 is a flowchart outlining the steps for generating an output image using the present techniques;

[043] Figure 2 shows a plurality of input images and example output images

[044] Figure 3 is a schematic drawing of some modules used in the method of Figure 1;

[045] Figure 4 is an overview of the hypernetwork and its use in generating merging coefficients in the method of Figure 1;

[046] Figures 5a and 5b show examples of merging coefficients generated using ZipLoRA for randomly selected columns of the LoRA weight update matrices;

[047] Figure 6a is a flowchart showing more details of the steps of generating the image using the merged adapter and in particular details how the text prompt is used;

[048] Figure 6b is a schematic diagram of the modules for generating an image using the method of Figure 6a;

[049] Figure 7 is a flowchart showing the steps of one method for assessing the generated image;

[050] Figure 8 is a flowchart of the steps of building a dataset of LoRAs;

[051] Figure 9a is a flowchart showing the hypernetwork training;

[052] Figure 9b shows an algorithm which is suitable for the training of Figure 9a;

[053] Figure 9c shows an algorithm which is suitable for the merging steps carried out by the hypernetwork in the inference process in Figure 1;

[054] Figure 10 is a system diagram of the server and user device; and

[055] Figures 11a and 11b show examples of merging coefficients generated using the present techniques for randomly selected columns of the LoRA weight update matrices; Detailed description of the drawings

[056] Broadly speaking, the present techniques generally relate to a method and apparatus for generating personalized images. In particular, the present techniques provide an image generation system which generates an output image based on user selected content and style as well as a method for training the image generation system. Advantageously, the present techniques, the present techniques include the use of adapters which are combinable with the image generation model to personalize the output image. The present techniques also include a hypernetwork to produce weight coefficients for merging subject / content and style adapters. The present techniques optionally further comprise an automatic evaluation protocol of the generated image based on a multimodal large language model.

[057] Figure 1 is a flowchart of the method of deploying the present techniques to generate, using a hypernetwork, good quality merging coefficients on the fly for new unseen content and style LoRAs so that good quality images may be generated by the image generation system. In the first steps, three inputs are received, and these may be received simultaneously as shown or in any order. At step S102, a text input is received. At step S104, a content input is received and at step S106, a style input is received. Styles may comprise specific / recognisable design patterns, textures, colours or material. The inputs may be received from a user, e.g. via a suitable interface on the device. The style and content inputs may be in the form of images representing the style or content or may be in the form of LoRAs which have been adapted to handle the style and content.

[058] Low-Rank Adaptation (LoRA) is described, for example in “Low-rank adaptation of large language models” by Hu et al. published in 2021 or “ZipLoRA: Any subject in any style by effectively merging loras” by Sohn et al published in European Conference on Computer Vision in 2024. LoRA is a technique to adapt machine learning models (which may be termed base models) to new contexts by adding lightweight pieces to the original model rather than changing the entire model. In the context of image generation, LoRA models may be small, fine-tuned versions of stable diffusion models which are then added to the base model, which is an image generation diffusion model. Each LoRA model (which may also be termed an adapter) has a set of parameters which are trainable. The set of parameters for the LoRA model is significantly smaller than the set of parameters for the base model. This may be expressed as using an image generation diffusion model 2) with weights Wo and LoRA parameters L with weight update matrix bW. A model 2) that uses a LoRA L is denoted as 2\ = 2) © L with weights Wo + bW, where operation © means the LoRA L is applied to the base model 2). LoRA can be applied to any subset of weight matrices in a neural network to reduce the number of trainable parameters. The weight matrices may comprise some or all of query weights Wq, key weights V / k, value weights Wv and output weights Wo. As explained below, LoRA may be applied to one or more of these different types of weights.

[059] Examples of content and style inputs are shown in Figure 2. Each of these inputs is a sample image. The content input is an image for a reference subject (e.g. my dog, my cat or my clock). The style input is an image which represents a reference style (e.g. an image representing a van Gogh painting or an image representing Japanese anime). The text input may be a textual prompt for generating the personal subject in the personal style (e.g. generate an image of my dog in the style of a van Gogh painting). The text input may include prompts to specify more details for the output image, e.g. to specify an action for the personal subject. Merely as examples, the text prompt may be “my dog playing with a ball in the style of a van Gogh painting”, “my dog sleeping in the style of a van Gogh painting” or “my dog wearing a crown in the style of a van Gogh painting”.

[060] Style-conditioned generation is described for example in “A style-based generator architecture for generative adversarial networks” by Karras et published in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2019 and termed StyleGAN, “StyleDrop: Text-to-image synthesis of any style” by Sohn et al. published in Advances in Neural information Processing Systems in 2023 and “DreamArtist: Towards controllable one-shot text-to-image generation via positive-negative prompt-tuning” by Dong et al. published in arXlv in 2022. However, these methods lack the ability to handle both subject and style conditioning jointly.

[061] “HyperDreamBooth: Hypernetworks for Fast personalization of Text-to Image models” by Ruiz et al published in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2023 generates personal subjects with good style editability via textual prompt. This paper designs a hypernetwork to efficiently generate model weights based on a single reference image. Hypernetworks have also been used in other use cases such as generating LoRA parameters as described in “A study of parameter efficient fine-tuning by learning to efficiently fine-tune” by Ceritli et al. that was presented in the Findings of the 2024 Conference on Empirical Methods in Natural Language Processing. A hypernetwork is a network which generates the parameters of other networks (e.g. the LoRA). In the present techniques, the hypernetwork is used to predict merging coefficients for subject-style LoRA fusion, enabling efficient, high-quality joint personalization without extensive optimization overhead.

[062] Returning to Figure 1, when the content and style inputs are in the form of images, corresponding content and style LoRa adapters are obtained at steps S114 and S116. It will be appreciated that if the content and style inputs are in the form of the LoRAs, these steps can be omitted. The LoRAs can be obtained from a suitable marketplace for LoRAs such as, for example, from Samsung, Hugging Face or Civitai amongst others. As explained below, a dataset of LoRAs may be built as part of the training process and the corresponding adapters may be obtained from this dataset. As noted above, a model 2) that uses a LoRA L may be denoted as DL = D ® L with weights Wo + &W. To specify content and style, two separate LoRAs Lc (content LoRA) and Ls (style LoRA) with respective weight update matrices &WC and are used.

[063] Optionally, the pair of style and content adapters may be stored on device as shown at step S120. The storage may be temporary. As another optional step, the LoRA adapters may be fine-tuned using the user’s own images as shown at step S122. Suitable techniques for fine-tuning the LoRA adapters are described below and include the Dreambooth protocol which is described for example in “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation” by Ruiz et al published in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2023. However, storage and / or fine-tuning increase the requirements on the user device and may be skipped.

[064] At step S124, the two adapters (with / without fine-tuning) are input to the hypernetwork to predict the merging coefficients so that a merger adapter can be generated at step S126. Using the terminology above, the objective is to merge Lc and Ls into Lm to obtain a merged adapter, producing a matrix AWm that combines with the parameter matrix for the generative ML model whereby the required / desired content and style is coherently combined in generated images. At step S130, the output image is then generated using the merged adapter together with the base model. The base model may be any suitable diffusion model such as the Stable Diffusion model XL model which is described for example in “SDXL: Improving latent diffusion models for high-resolution image synthesis” by Podell et al. published in ArXiv in 2023. As shown, the text prompt may be used in the generation of the image as described in more detail below. Optionally the generated image is then assessed at step S132 using a multi-model large language model (MLLM) as described in more detail below or alternatively using human judgement. The generated image is then output at step S134. Returning to Figure 2, examples of the output images are shown in the rows adjacent the style image.

[065] Figure 3 schematically illustrates the generation of the merged adapter. An input content image 300 and an input style image 302 are received as described above. The content adapter 310 and style adapter 312 which correspond to these inputs are then obtained. The obtained adapters are inputs to the hypernetwork 320 which outputs the merging coefficients. The content adapter 310 and the style adapter 312 are then merged using these merging coefficients to create a merger adapter 322. The merged adapter is input into the base model so that the output image shows the content in the style. In this example, an image of my dog is the content, an image of a van Gogh painting is the style and the output is an image of my dog in the style of a van Gogh painting.

[066] Figure 4 provides more detail on the merging of the two adapters. As shown, the hypernetwork 420 which may be denoted by y or (and the terms are used interchangeably) takes two LoRA adapters or update matrices 410, 412 as inputs. A first update matrix 410 is for content and is represented by &WC e ]Rmxn and a second update matrix 412 is for style and is represented by AM4 e ]Rmxn. The method predicts the column-wise merging coefficients mc e HR" and ms e HR". Given the high dimensionality of each update matrix, flattening them directly as input would be impractical. To address this, it is assumed that the merging coefficient for each column can be predicted independently.

[067] For each column i, the respective content and style columns are extracted as denoted by: wj, = Wc[:, j] and = M4[:, j], and are concatenated as [wlc, w|] e IR2’" to form the input features for the hypernetwork. Different columns are treated separately as a minibatch, allowing for efficient parallel processing. The full hypernetwork input 416 may be expressed as: conca^AW^J, AVZJ, dim = 1) e [R"x2m To accommodate the various LoRA matrix sizes within the diffusion model 2), the hypernetwork is designed with separate input layers tailored to each unique matrix size, each mapped to a shared hidden dimension. As shown in the example in Figure 4, the hypernetwork uses two input layers with ReLLI non-linearities and a shared output layer to predict merging coefficients for each column. In this example, an example layer with dimensions matching input layer 1 is shown. The image generation backbone (base model) has a plurality of layers which may have the same or different dimensions. The number of input layers in the hypernetwork may be tailored by counting the number of unique layer shapes in the image generation backbone.

[068] Since different rows are treated as a mini-batch, overall the hypernetwork outputs 2n coefficients, one for each column of content and style LoRAs: mc,ms = J-C(Lc,Ls~) These coefficients are used to merge the LoRAs Lc and Ls, resulting in the merged LoRA Lm with update matrix AM^: AW^ = mc ® AM4 + ms ® AU2

[069] Notably, the hypernetwork-guided merging is used for query and output LoRAs, while simple averaging may be used for key and value LoRAs. This configuration empirically outperformed other tested options, as detailed below.

[070] It is noted that ZipLoRA takes a gradient-based approach, learning column-wise merging coefficients mc and ms for &WC and AM4, respectively, as: A Pl2,., = mc ® &WC + ms ® AM2, where ® represents element-by-column multiplication. Although ZipLoRA achieves high-quality results, it requires training these coefficients from scratch for each content-style pair, with distinct coefficients for different combinations. With ZipLoRA performing 100 gradient updates per pair, real-time performance is unfeasible, particularly on resource-constrained devices. Figures 5a and 5b shows examples of merging coefficients generated using ZipLoRA for randomly selected columns of the LoRA weight update matrices. The coeffiicents are visibly different for various combinations of content and style scenarios (e.g. for teapot in the style of a flat cartoon v dog in the style of a 3D rendering), showing the need for alternative solutions such as the present techniqes.

[071] Figure 6a is a flowchart showing more detail of the steps of generating the image using the merged adapter and in particular details how the text prompt is used. Figure 6b shows the key components in the image generation system, namely the text decoder 610, the diffusion backbone model 620, an image decoder 630 and a noise generator 640. Returning to Figure 6a, as shown at step S600, a text input is received as described in relation to Figure 1. The text input is processed using a text encoder in the image generation system to generate text embeddings at step S610. Any suitable text encoder, e.g. CLIP (proposed by Radford et al. “Learning Transferable Visual Models From Natural Language Supervision” that was presented in the International Conference on Machine Learning in 2021), may be used to generate any suitable text embeddings. The text embeddings are a representation of the text input and may be expressed as a vector of tokens. The embedding vectors may be transformed into a single conditioning code which guides the generative model. The embedding vector may be optimized with a pseudo-word using a reconstruction objective. At the same time (or before / after) generating the text embeddings, a latent representation of the image to be generated is also generated. A latent representation of the image is a lowerdimensional representation or embedding (e.g. an embedding vector) of the image data. The term “latent” is used because the representation captures hidden, underlying features of the data which may not be directly observable from the input space. Gaussian noise is obtained at step S602 and used as a latent seed to generate the latent representation using standard techniques at step S604. A merged adapter is generated at step S618 as described above.

[072] At step S620, the latent image representation and the text embeddings are input to the combination of the base model with the merged adapter. The base model may be a diffusion backbone model described in High-Resolution Image Synthesis with Latent Diffusion Models by Rombach et al. that was presented in the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2022. The model may use a UNet architecture proposed by Ronneberger et al. in “U-Net: Convolutional Networks for Biomedical Image Segmentation” that was presented in the International Conference on Medical Image Computing and Computer-Assisted Intervention in 2015. By inputting the latent image representation with the text embeddings, the latent image representation may be considered to be conditioned via the text embeddings. At step S622, a conditioned latent image representation is output. There is a determination at step S624 as to whether the required number of steps / iterations T have been carried out. If there are more iterations, a reconstruction algorithm is applied at step S626 to generate an updated latent image representation. The input and output steps of S620 and S622 are then repeated with the updated latent image representation. In other words, the base model (UNet) iteratively denoises the latent image representations, while being conditioned via the text embeddings from step S610. After the required number of iterations has been done, the current conditioned latent image representation is output at step S628 to a decoder (e.g. a variational autoencoder decoder) for conversion into the output image. At step S630, the generated image which combines the content, style and text prompt is output.

[073] Figure 7 is a flowchart showing the steps of one method for assessing the generated image (e.g. step S132 of Figure 1). Developing reliable metrics that align with user preferences is useful for scaling text-to-image models, especially when direct feedback is unavailable. Metrics like CLIP-I, CLIP-T, and DINO which are described for example in “Dreambooth: Fine tuning text-to-image diffusion models for subject-drive generation” by Ruiz et al. published in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2023 are widely used for single-concept personalization (i.e. personalizing to either a style or subject). However, these metrics may not reliably evaluate joint subject-style personalization,. Specifically, the CLIP-I score tends to favor style fidelity, often overlooking an accurate representation of the subject, while the DINO score prioritizes the original subject replication overlooking stylistics integration such as morphing, modifying in texture or “cartoonisation”. CLIP-T, typically used for text alignment, supports subject recontextualization but is less suited to style-content prompts like “A [c] <class name> in [s] style”. Here [c] is a unique rare token identifier for content, <class name> is the class name, for example as set out in the Dreambooth paper above, and [s] is a short description of the style, for example as in “StyleDrop: Text-to-image synthesis of any style” by Sohn et al. published in Advances in Neural information Processing Systems in 2023.

[074] To overcome the limitations of conventional metrics, as shown in Figure 7, MLLMs are proposed for evaluation. LLMs have shown high effectiveness in evaluating text-based outputs, and their application has recently been extended to multimodal tasks involving both text and images as described for example in “MLLM-as-a-judge”: Assessing multimodal LLM-as-a-judge with vision-language benchmark” by Chen et al. published in ArXiv in 2024. Recently specialized MLLM judge models have been developed, in particular the LLaVA-Critic which is described in “LLaVA-Critic: Learning to evaluate multimodal models” by Xiong et al. published in 2024. This model has been trained to accurately rate the quality of the outputs in multimodal contexts.

[075] In a first step S700 of Figure 7, the output image is generated, for example as described at step S130 of Figure 1. In this work, LLaVA-Critic is used to evaluate whether generated images accurately represent the intended subject (content) and style, although it will be appreciated that other comparable MLLMs may also be used. The protocol is as follows: the MLLM judge first assesses if each generated image meets the specified style and content independently as shown by the two separate branches in Figure 7. In other words, a prompt for assessing whether the style in the generated image is correct is generated at step S704 and similarly, a prompt for assessing whether the content in the generated image is correct is generated at step S706.

[076] Examples of the prompts used in steps S704 and S706 are set out below. [Prompt for assessing if correct subject is generated] Your task is to identify if the test image shows the same subject as the support image. Support image: {Image} Test image: {Image} Pay attention to details of the subject, it should for example have the same colour. However, the generated style of the image may be different. Does the test image show the same subject as the support image? Answer with yes or no only. [Prompt for assessing if correct style is used in the generated image] Your task is to identify if the test image shows the subject in {style} style. An example image in the {style} style is provided. Example image in the {style} style: {Image} Test image: {Image} The example image shows an illustration of the {style} style and the details of the subject are expected to be different. Do not check similarity with the subject. Is the test image in the {style} style? Answer with yes or no only.

[077] For clarity, we provide reference images for both style and content along with detailed evaluation prompts. Thus, as shown at step S714, the content prompt together with at least one content image are input to the MLLM and an assessment is made at step S724 as to whether there is a match in terms of content only to obtain an output “yes” or “no”. Similarly, at step S716, the style prompt together with at least one style image is input to the MLLM and an assessment is made at step S726 as to whether there is a match in terms of style only to obtain an output “yes” or “no”. Binary ratings are used for both style and content evaluations. These binary ratings are combined as shown at the matching step S728, with an image deemed correct, final score of 1, only if it fulfills both criteria, and 0 otherwise. If one or both criteria are not met, the image is rejected at step S730. If both criteria are met, the image may be output at step S732.

[078] When there are multiple reference images, a generated image may be considered to be accurate if the MLLM model identifies it as correct for more than half of the reference images. This new metric may be termed MARS: Multimodal Assistant Rating Subject&Style. For each content-style pair, the method may generate multiple images and evaluate both the average and best sample quality according to the MLLM judge. This dual evaluation provides flexibility for users in downstream applications, allowing them to select the most preferred sample.

[079] In addition to evaluation via MLLMs, human evaluation on a subset of the generated examples is also used to assess performance because assessing all generated image manually is not viable. In the human evaluation, there is an assessment as to whether the generated images are better, similarly good or worse than those generated by the relevant baselines such as ZipLoRA and simple arithmetic merge as detailed below. Two cases are considered separately. In the first case, one generated sample from each approach is randomly selected for every test content-style LoRA pair and in the second approach a best sample is taken. For unbiased feedback, method names are anonymised. This human evaluation offers insights into real-world user preferences and serves as qualitative validation of the proposed approach.

[080] As shown in Figure 8, before training the hypernetwork, a dataset of LoRAs is first built. Content LoRAs are trained on individual subjects from the Dream Booth dataset, and style LoRAs are trained on various styles from the StyleDrop / ZipLoRA datasets. Each LoRA is generated using the DreamBooth protocol. The LoRA dataset is split into training {jLtram^ validation and test {JL^est}, {Lsest} sets. During training and evaluation, content-style LoRA pairs are sampled. The hypernetwork is trained on the training sets, with hyperparameters and design choices tuned on the validation sets. The test sets are reserved to assess performance on novel content-style pairs.

[081] Figure 8 is a flowchart showing the steps for training adapters to include in the dataset of LoRAs. At step S802, the user input is received as described above in relation to the inference stage. In other words, the user inputs images depicting the content and style to be combined in the image generation and a prompt. At step S804, a suitable base model is obtained, for example a diffusion model as described above. This base model does not include the LoRA adapters.

[082] In step S806 there is an optional step to set the rank for the LoRAs which are to be used. It will be appreciated that the rank may already be determined. In LoRA, the weight update matrix bW can be represented as the product of two smaller matrices A and B each of which is of lower dimensionality than weight matrix W of the base model. By choosing matrices A and B to have a lower rank r, the number of trainable parameters is significantly reduced. For example, if W is a d x d matrix, traditionally, updating W would involve d2 parameters. However, with B and A of sizes d x r and r x d respectively, the total number of parameters reduces to 2dr, which is much smaller when r « d. The rank may be selected to provide a reasonable balance between the number of parameters to be updated and any loss in performance.

[083] The base model is already trained using standard techniques and is typically expensive to retrain, so only the LoRA adapters are trained. Thus, at step S814, the parameters of the base model are frozen and an adapter is obtained at step S816. The obtained LoRA is then combined with the base model and an image is generated from the combined model in step S820. A loss is calculated in step S822 (e.g. the loss from the Dreambooth protocol). The number of iterations is counted at step S824. When the number of iterations has reached a pre-determined k iterations then the trained LoRA is output at step S828. Otherwise, the LoRA adapter is updated (i.e. the parameters are adjusted to reduce the loss) at step S826 and another iteration is triggered starting at the image generation step S820.

[084] Figure 9a is a flowchart showing the hypernetwork training. At step S902, the general training parameters are obtained, these include the number of training steps t = 1,..., T which will be carried out together with the learning rate ij. At step S904 the base model is obtained, for example from any suitable database. At step S906 the training dataset for the LoRA adapters is obtained, for example by following the process in Figure 8 or obtaining a training dataset from any suitable database.

[085] In step S910 the iteration count is checked to see if the required number of training steps has been reached and the required number of training steps has been completed, the trained hypernetwork is output at step S930. For the first and subsequent iterations, each of the following steps is carried out to train the hypernetwork Jf. In a first step S912 of the iterative process, a content-style LoRA pair is sampled / selected from the training set {IL*™”1}, {I*1™"}. Each pair comprises a style LoRA and a content LoRA. The selected pair of LoRAs are input into the Hypernetwork, and a set of merging coefficients mc,ms for the LoRA adapters are predicted at step S914. At step S916, the predicted merging coefficients mc,ms are used by the Hypernetwork to merge the pair of LoRAs together to form a merged LoRA defined by: AWm = mc 0 AWC + ms 0 A

[086] . This merged LoRA is combined with the base model and an image is generated in step S918 in a similar manner to that used at inference. In other words, a sample content image which corresponds to the selected content LoRA and a sample style image which corresponds to the selected style LoRA are used in the training process. Text prompts may also be used together with noise to generate latent image representations. A loss may then be computed at step S920 based on the difference between the desired output image and the image which is generated in step S918. Using standard techniques, this loss may then be used to update the hypernetwork parameters in step S922.

[087] An example of a suitable loss which may be used in step S920 is a merging loss £merge that updates the parameters of . The merging loss £merge from the ZipLoRA paper above, which includes terms that ensure both content and style fidelity, while also encouraging orthogonality between content and style merging coefficients is a suitable loss. Specifically, Emergeis defined as: ■Emerge = H © ^m)Cxc> Pc) — © ^c)Cxc> Pc) H2 + II (2) ®Lm)(xs,ps) - (D ® Ls)(xs,ps) ||2+A|mc • ms| where xc,xs are the noisy latent image representations (see Figure 6), pc,ps are the text prompts for content and style reference images respectively, the term A controls the strength of the orthogonality-promoting regularization term, 2) is the base model, Lm is the merged adapter, Lc is the selected content adapter and Ls is the selected style adapter. Using this loss, the update to the parameters of the hypernetwork may be defined as: 2T Emerge

[088] After step S922, the iteration count is incremented and the algorithm moves back to step S910 to check whether the required number of training steps has been completed or there are more training steps to loop through. Once the trained hypernetwork is output, there may be further testing on the test set of LoRAs. The key goal of training the hypernetwork is create a hypernetwork which can generalise to new unseen content and style LoRAs so that they can be merged in real time and with superior quality.

[089] The objective is to design and train a hypernetwork that predicts weighting coefficients to merge content and style LoRAs. As explained above, using a set of LoRAs, this hypernetwork is trained to produce suitable merging coefficients for unseen content and style LoRAs at the deployment stage. The training process is formalized in the algorithm shown in Figure 9b. For completeness, the algorithm for the merging steps carried out by the hypernetwork in the inference process in Figure 1 is also shown in Figure 9c.

[090] Figure 10 illustrates an example system which may be used to implement the new techniques described above. The system comprises a server 1000 which may be a single server or collection of servers (e.g. the cloud). The server 1000 comprises at least one processor 1002 coupled to memory 1004. The at least one processor 1002 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The at least one processor 1002 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The memory 1004 may comprise volatile memory, such as random-access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example.

[091] An image generation model 1006 which comprises a base model (ML model) 1016, an MLLM 1014 and a hypernetwork 1018 may be stored on the server 1000, e.g. during the training described above. The server 1000 also comprises an input / output interface 408 (or similar communication module) which connects the device to a database 1010. The database 1010 may comprise training dataset(s) for training the ML model. The database 1010 may also comprise LoRAs which can be added to the base model as described above. The server 1000 is also coupled to at least one apparatus / user device 1020.

[092] The user device 1020 also comprises similar standard components to the server 1000. The user device 1020 comprises at least one processor 1022 coupled to memory 1024. The at least one processor 1022 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The at least one processor 1022 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The memory 1024 may comprise volatile memory, such as random-access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example. A copy of the image generation model 1026 which comprises a base model (ML model) 1036, a MLLM 1034 and a hypernetwork 1038 is stored on the electronic device 1020. It is possible to avoid storing the base model and just download the base model when required. User data 1030 which is used to personalize the model, e.g. to fine-tune LoRAs during the inference process. There is also an input / output interface 1028 which connects the user device 1020 to the server 1000.

[093] The proposed approach is compared to several established methods, including: joint training of both content and style via Dreambooth described in “Dreambooth: Fine tuning text-to-image diffusion models for subject-drive generation” by Ruiz et al. published in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2023” and direct merging of LoRA weights described in “Model soups: averaging weights of multiple fine-tuned models improves accuracy within increasing inference time” by Wortsman et al. published in International Conference on Machine Learning in 2022. General model merging techniques such as DARE described in “Language models are super mario: Absorbing abilities from homologous models as a free lunch” by Yu et al. publihsed in Forty-first International Conference on Machine Learning in 2024, TIES and DARE-TIES described in “Ties-merging: Resolving interference when merging models” by Yadav etal. published in Advances in Neural Information Processing Systems in 2024; and ZipLoRA described in “ZipLoRA: Any subject in any style by effectively merging loras” by Sohn et al. published in Conference on Computer Vision in 2024. As described above, ZipLoRA is specifically designed for merging subject and style LoRAs. ZipLoRA has also been compared with strategies such as StyleDrop, Custom Diffusion and Mix-of-show. These methods, however, have been shown to perform less effectively while being computationally costly, so are excluded from further comparison in this work.

[094] Implementation details: All experiments use the SDXL v1 Stable Diffusion model described in “SDXL: Improving latent diffusion models for high-resolution image synthesis” by Podell et al. published in ArXiv in 2023, following the setup in the ZipLoRA paper. For subjectspecific LoRAs, rare unique token identifiers are adopted as described in the Dreambooth paper. The style LoRAs are fine-tuned using text description identifiers instead of a rare token, following the technique described in the Styledrop paper, where these were found more effective for style representation.

[095] Base LoRAs are trained as described in Figure 8, for 1000 fine-tuning steps, with batch size 1, a learning rate of 5 x 10-5 and a rank of 64. The text encoder remains frozen during training. The hypernetwork used is a two-layer MLP with two separate input layers of size 1280 and 2560, followed by a ReLLI activation function, a shared hidden layer of size 128, and two outputs. The hypernetwork is trained as described in Figure 9 for 100 different {LC,LS} combinations (totalling 5000 steps), with A = 0.01, learning rate 0.01 and the AdamW optimizer. For ZipLoRA, a training setup of 100 steps with the same A and learning rate. The DARE, TIES, and DARE-TIES baselines are evaluated with uniform weights and a density of 0.5. For joint training, a multi-concept variant of Dreambooth LoRA was used as in the ZipLoRA paper. In all experiments, 50 diffusion inference steps are used.

[096] Datasets. The dataset for training the hypernetwork is not composed of images, but is a set of LoRAs. Subject LoRAs are fine-tuned on images from the DreamBooth dataset and style LoRAs are fine-tuned on images from the StyleDrop and / or ZipLoRA datasets. Specifically, the datasets includes 30 subjects (each with 4-5 images) and 26 styles (each represented by a single image). Fortraining, validation, and testing, the subjects are split into 20-5-5 items for training, validation and testing and the styles into 18-3-5 items for training, validation and testing. The subjects and styles are split randomly, but with the constraint that there is a good representation of different subjects and styles in each split as some subjects and styles are similar to each other. For example, it is aimed to avoid only testing on different dogs or only on painting styles. This splitting yields a total of 360 subject-style LoRA combinations for hypernetwork training, a quantity shown to be sufficient for robust performance. Merely as examples, the 20 training concepts may include backpack, backpack dog, berry bowl, candle, cat, colourful sneaker, dog, dog 5 dog 6, dog 7, duck toy, fancy boot, grey sloth plushie, monster toy, pink sunglasses, poop emoji, rc car, red cartoon, robot toy, shiny sneaker and vase. Similarly, merely as examples, the 18 training styles may include 3D rendering, 3d rendering 3, abstract rainbow, black statue, cartoon line drawing, flat cartoon illustration, glowing 3D rendering, kid crayon drawing, line drawing, melting golden rendering, oil painting 3, sticker, watercolour painting 2, watercolour painting 4, watercolour painting 5, watercolour painting 6, watercolour painting 7, wooden sculpture. The test concepts may be dog 8, cat 2, wolf plushies, teapot and can. The test styles may be 3d rendering 4, oil painting 2, watercolour painting 3, flat cartoon illustration 2, glowing. The validation concepts may be dog 2, dog 3, clock, bear plushie. The validation styles may be 3d rendering 2, oil painting and watercolour painting.

[097] Evaluation Details. For the MLLM-based metric, MARS2, the LLaVA-Critic 7b model is used as described above. The prompts used for the MLLM model are detailed above. The table below shows the ratio of generated images with the correct content and style on the combinations of test subjects and styles according to the new metric MARS2. Average case Best case Joint Training 0.53 0.84 Direct Merge 0.40 0.76 DARE 0.34 0.72 TIES 0.43 0.80 DARE-TIES 0.30 0.60 ZipLoRA 0.58 1.00 LoRA.rar 0.71 1.00

[098] Merely for comparison, the various methods are also compared using standard metrics and the results are shown in the table below. As explained above, these metrics are not optimal for the joint subject-style personalization task: CLIP-I DINO CLIP-T Joint Training 0.623 0.764 0.329 Direct Merge 0.657 0.747 0.305 DARE 0.630 0.576 0.360 TIES 0.620 0.592 0.358 DARE-TIES 0.618 0.559 0.355 ZipLoRA 0.643 0.741 0.334 LoRA.rar 0.656 0.643 0.344

[099] The proposed solution termed LoRA.rar consistently outperforms all methods, including ZipLoRA, in both content and style accuracy. For the best sample (selected by MLLM from 10 generated images as one with correct style and content if available), both the proposed solution and ZipLoRA achieve perfect accuracy, indicating that users can reliably choose 15 preferred outputs when multiple samples are available. Across all generated images, the proposed solution performs better on average than ZipLoRA, likely benefitting from its capacity to leverage knowledge learned from diverse content-style LoRA combinations.

[100] For human evaluation, evaluators are presented with content and style reference images alongside randomly ordered outputs from each method. Each of the 25 evaluators assesses 25 pairs of images comparing two approaches. The pairs of images are generated by either the proposed approach or ZipLoRA, and they are randomly ordered in each pair. The two approaches are, one using randomly generated images and one using the “best” images as judged by the MLLM judge. In the “best” scenario, all the images that satisfied both subject and style according to the MLLM judge are gathered and then selected one randomly among those-there was always at least one such example for each approach.

[101] An example task preceded the evaluation to clarify the assessment criteria. The evaluators were given the textual instruction: “Your task is to evaluate which of two generated images better represents the given subject and style - or if they are similarly good. You are provided with an image showing the subject (e.g. black cat) and an image showing the image style (e.g. van Gogh style painting), and two generated images such as in the example below. In this example you would select option 2 as better because it shows a cat that looks like the one in the subject image, and both images follow the style." The evaluation was done via a web app that shows the images and lets the participant click on a button saying which option is better among: “Option 1", “Similar”, “Option 2”. The table below shows the summary results of this analysis: LoRA.rar better Similar ZipLoRA better Random 47.4% 36.8% 15.8% Best 48.8% 28.2% 23.0% This evaluation was conducted on a subset of generated samples as described earlier and focused on ZipLoRA as the primary comparison baseline, given the time constraints of manual assessment. The results indicate that the proposed solution compares favorably with ZipLoRA, confirming that the generated images are typically either better or comparable in quality. Furthermore, the proposed solution can operate in real-time for new subject-style combinations, unlike ZipLoRA.

[102] A qualitative analysis of the proposed method was also conducted by: (1) comparing the images generated by LoRA.rar with those produced by competing methods, and (2) analyzing the diversity of images generated by LoRA.rar across concepts and styles. The results demonstrate that LoRA.rar excels in capturing fine details across various styles, consistently producing high-quality images. While ZipLoRA also generates high-quality images, LoRA.rar outperforms it in terms of overall fidelity to both content and style. A limitation of ZipLoRA is in too realistic generation, for example a teapot in 3D rendering style is immersed in a photorealistic scene, and a wolf plushie in oil painting is not a painting. Other approaches show less consistent results, for example direct merge is able to produce a teapot or a stuffed animal in 3D rendering style (with minor inaccuracies), but fails at generating flat cartoon illustrations, where there is no one-to-one mapping of content and style. DARE, TIES, DARE-TIES do not produce satisfactory results, either the style or the concept are incorrect, or none. Joint training, presents improved results compared to direct merge, but has the same limitations. LoRA.rar consistently works well across diverse combinations of concepts and styles, highlighting its versatility and effectiveness.

[103] Additional analyses: A detailed analysis of resource usage is provided in the table below which highlights the efficiency and scalability of LoRA.rar in comparison to ZipLoRA. A “good” image is one which is accepted by MARS (i.e. the new MLLM-based evaluation protocol) and when determining the number of parameters for ZipLoRA, only one subject-style pair is considered.: ZipLoRA LoRA.rar (ours) Time to predict merging coeffs 158s 0.037s # Parameters 1.5M 0.49M # Attempts to good* image 2.55 2.28 Extra memory at test time 4GB 0GB

[104] 1) Runtime Efficiency: As shown in the table above, the proposed solution generates the merging coefficients over 4,000 times faster than ZipLoRA on an NVIDIA 4090 while using 3 times fewer parameters (despite adding a hypernetwork) and achieves real-time performance. While ZipLoRA requires 100training steps for each concept-style pair, LoRA.rar generates merging coefficients in a single forward pass (per layer) using a pre-trained hypernetwork. Similarly, on an NVIDIA 3090, it takes around 5 minutes to do an inference using ZipLoRA. The proposed techniques need a single forward pass to predict the merging coefficient with the trained hypernetwork (0.037 seconds).

[105] 2) Parameter Storage: ZipLoRA needs to store the learned coefficients for every combination of concept and style for later use. LoRA.rar only needs to store the hypernetwork, which has 3 times fewer parameters than a single ZipLoRA combination.

[106] 3) Sample Efficiency: on average, LoRA.rar requires fewer attempts than ZipLoRA to produce a high-quality image that aligns with both content and style - 2.28 attempts for LoRA.rar versus 2.55 for ZipLoRA. This improvement reflects LoRA.rar’s enhanced accuracy in generating visually coherent outputs without extensive retries, further optimizing resource usage and user experience.

[107] 4) Memory Consumption at Test Time: LoRA.rar is efficient in terms of memory, which is dominated by the generative model (-15GB), with negligible overhead for our approach, while ZipLoRA requires additional 4GB (-19GB totally).

[108] Finally, the merging coefficients learned by LoRA.rar are analysed in Figures 11a and 11b. LoRA.rar learns a non-trivial adaptive merging strategy, with diverse coefficients (including also some negative values). This adaptability allows LoRA.rar to flexibly combine content and style representations, likely contributing to its superior performance. ZipLoRA, instead, mostly converges to an adaptive yet binary merging strategy (see Figures 5a and 5b). This more rigid merging strategy may limit its capacity to finely integrate details across styles and subjects, further underscoring the advantage of LoRA.rar’s approach in generating images with accurate content and style.

[109] Ablation Study on Hypernetwork - An ablation study was also conducted on the hypernetwork design by exhaustively exploring all possible configurations to determine which components should have their merging coefficients predicted by the hypernetwork The validation set and MLLM judge were used for this investigation, and the results are reported in the table below. The best results are obtained by Query, Output case and these have been used as described above. However, a few other combinations also achieve good results such as Query, Key and Output as well as Query, Value and individually Value. Average case Best case Key 0.28 0.75 Value 0.43 0.83 Query 0.28 0.75 Output 0.39 0.83 Key, Value 0.40 0.92 Key, Query 0.31 0.75 Key, Output 0.44 0.75 Query, Value 0.42 0.83 Query, Output 0.48 0.92 Value, Output 0.29 0.58 Query, Key, Value 0.41 0.83 Query, Value, Output 0.23 0.33 Query, Key, Output 0.49 0.83 Key, Value, Output 0.29 0.50 Query, Key, Value, Output 0.23 0.50

[110] As demonstrated above, the present techniques achieve comparable or superior quality with respect to ZipLoRA while enabling real-time merging (i.e. in under a second) which would make such technology far more accessible for deployment on lightweight devices. In summary, the present techniques include training a simple hypernetwork to learn merging coefficients for arbitrary subject and style LoRAs. The hypernetwork is trained on a small dedicated (i.e. curated) dataset of LoRAs. At test time, two input LoRAs are fed into the hypernetwork and the merging coefficient is predicted and output on this unseen test set. The main beneifit may be generalization to unseen subject-style combinations, no test-time training required (in other words generating merging coefficients instantly via a single forward pass and thus removing the need for retraining) and superior results to both simple merge and ZipLoRA. Additionally, a new metric based on a multimodal large language model (MLLM) is proposed and to verify that the generated image is more aligned to the human preference.

[111] In summary, the present techniques make use of the availability of large numbers of LoRAs and use LoRA weights as training data. There is a training / validation / test split to evaluate performance. A simple HyperNetwork (which is small and has 0.5 million parameters) is proposed which is given as an input a pair of concept and style LoRAs and predicts the merging coefficients that are used when merging the LoRAs. The approach is fast and lightweight and can run in real-time (i.e. less than 0.1 second) on a server or a resource-constrained device such as a mobile device. The approach generalises seamlessly to new LoRAs without the need for further test-time training. The limitations of the current metrics in assessing the simultaneous generation of a specific concept in a specific style are also explored. A new MLLM-based evaluation protocol is proposed which is highly aligned to user preference (this may be termed MARS as detailed above). This can help scale the quantitative studes beyond expensive and time consuming user studes and the ineffective state-of-the art metrics. In cases where users prefer to input images rather than LoRAs, image-to-LoRA encoding methods like those described in DiffLoRA (“DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion” by Wu et al. in arXiv in 2024) can transform reference images into LoRAs. 5

[112] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will 10 recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

1. A computer-implemented method for real-time generation of a personalized image usinga generative machine learning, ML, model, the method comprising:receiving, from a user, content input indicating content to be included in the generated image;receiving, from a user, style input indicating a style for the generated image;obtaining a content adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image containing the indicated content;obtaining a style adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image in the indicated style;predicting, using a hypernetwork, merging coefficients for merging the obtained content adapter and the obtained style adapter;merging, using the predicted merging coefficients, the obtained content adapter and the obtained style adapter to form a merged adapter;combining the merged adapter with the generative ML model to adapt the generative ML model to generate an image containing the indicated content in the indicated style; andgenerating, using the combined generative ML model and merged adapter, the image comprising the indicated content in the indicated style.

2. The method of claim 1, wherein receiving the content input comprises receiving an image of the content and receiving the style input comprises receiving an image which is in the indicated style.

3. The method of claim 1 or claim 2, wherein obtaining the style and content adapters comprises obtaining low-rank adaptation, LoRA, adapters each having weight update matrices.

4. The method of claim 3, wherein predicting merging coefficients comprises predicting merging coefficients by:extracting a column in the weight update matrix for the content adapter;extracting the corresponding column in the weight update matrix for the style adapter;concatenating the extracted columns andrepeating the extracting and concatenating steps for each column in the weight update matrices.

5. The method of claim 4, wherein predicting merging coefficients further comprises:inputting the concatenated columns into the hypernetwork; andoutputting a plurality of merging coefficients which comprise a merging coefficient for each column of each content adapter and a merging coefficient for each column of each style adapter.

6. The method of claim 5, wherein the hypernetwork comprises a plurality of input layers and inputting the concatenated columns into the hypernetwork further comprises:combining the concatenated columns to form a concatenated matrixselecting an input layer in the hypernetwork based on a size of the concatenated matrix; andinputting the concatenated matrix into the selected input layer of the hypernetwork.

7. The method of any one of claims 3 to 6, wherein the weight update matrices comprise query weights, key weights, value weights and output weights, andpredicting merging coefficients comprises:predicting merging coefficients for one or more of the query weights, key weights, value weights and output weights.

8. The method of claim 7, further comprising:determining whether merging coefficient have been predicted for each of the query weights, key weights, value weights and output weight; andwhen it is determined that merging coeffiicents have not been predicted, using averaged merged coefficients.

9. The method of any one of the preceding claims, further comprising:receiving, from a user, a text input indicating both the content and the style for the generated image.

10. The method of claim 9, wherein generating the image, using the combined generative ML model and merged adapter, comprises using a generative ML model in the form of a text-to-image generative ML model to generate the image using the received text input.

11. The method of claim 10, wherein the text-to-image generative ML model is in the form of a diffusion ML model and generating the image, using the combined generative ML model and merged adapter, further comprises:generating, using a text decoder, a text embedding of the text input;inputting, into the diffusion ML model, the generated text embedding together with noise as a latent seed to obtain an initial latent image representation;iteratively denoising the initial latent image representation to obtain an output latent image representation; andconverting, using a decoder, the output latent image representation to the generated image.

12. The method of any one of the preceding claims, further comprising, after generating the image:generating a first prompt for assessing whether the content in the generated image matches the indicated content;generating a second prompt for assessing whether the style of the generated image matches the indicated style;inputting the first and second prompts and the generated image into a multimodal large language model, MLLM; andoutputting from the MLLM an evaluation of whether the generated image accurately represents the indicated content in the indicated style.

13. The method of claim 12, wherein outputting an evaluation from the MLLM comprises: outputting, in response to the first prompt, a first assessment of whether the content in the generated image matches the indicated content;outputting, in response to the second prompt, a second assessment of whether the style of the generated image matches the indicated style; andoutputting an evaluation that the generated image accurately represents the indicated content in the indicated style when the first assessment indicates that there is matching content and the second assessment indicates that there is a matching style.

14. The method of any one of the preceding claims, further comprising, before using the generative ML model, training the hypernetwork by:selecting, from a dataset of adapters, a pair of adapters wherein each pair comprises a content adapter and a style adapter;predicting, using the hypernetwork, merging coefficients for the selected pair of adapters;merging, using the predicted merging coefficients the selected pair of adapters to form a merged adapter;combining the merged adapter with the generative ML model;generating, using the combined generative ML model and merged adapter, an image comprising content corresponding to the selected content adapter in the style of the selected style adapter; andupdating the parameters of the hypernetwork to minimize a merging loss for the generated image.

15. A computer-implemented method for training a hypernetwork to be used with a generative ML model and a pair of adapters for real-time generation of a personalized image, the training comprising:selecting, from a dataset of adapters, a pair of adapters wherein each pair comprises a content adapter and a style adapter, the content adapter is combinable with the generative ML model to adapt the generative ML model to generate an image containing content corresponding to the selected content adapter and the style adapter is combinable with the generative ML model to adapt the generative ML model to generate an image in a style corresponding to the selected content adapter;predicting, using the hypernetwork, merging coefficients for the selected pair of adapters;merging, using the predicted merging coefficients, the selected pair of adapters to form a merged adapter;combining the merged adapter with the generative ML model;generating, using the combined generative ML model and merged adapter, an output image comprising content corresponding to the selected content adapter in the style of the selected style adapter; andupdating the parameters of the hypernetwork to minimize a merging loss for the generated image.

16. The method of claim 14 or claim 15, further comprising:generating, using the combined generative ML model and content adapter, a first image;generating, using the combined generative ML model and style adapter, a second image; andcalculating a merging loss from a sum of a difference between the generated output image and the first image and a difference between the generated output image and the second image.

17. A user device for real-time generation of a personalized image using a generative machine learning, ML, model, the user device comprising:at least one processor andmemory storing instructions that, when executed by the at least one processor individually or collectively, cause the user device to generate a personalized image by: receiving, from a user, content input indicating content to be included in the generated image;receiving, from a user, style input indicating a style for the generated image;obtaining a content adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image containing the indicated content;obtaining a style adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image in the indicated style;predicting, using a hypernetwork, merging coefficients for merging the obtained content adapter and the obtained style adapter;merging, using the predicted merging coefficients, the obtained content adapter and the obtained style adapter to form a merged adapter;combining the merged adapter with the generative ML model to adapt the generative ML model to generate an image containing the indicated content in the indicated style; andgenerating, using the combined generative ML model and merged adapter, the image comprising the indicated content in the indicated style.

18. A server comprising:at least one processor andmemory storing instructions that, when executed by the at least one processor individually or collectively, cause the server to train a hypernetwork to be used with a generative ML model and a pair of adapters for real-time generation of a personalized image, by:selecting, from a dataset of adapters, a pair of adapters wherein each pair comprises a content adapter and a style adapter, the content adapter is combinable with the generative ML model to adapt the generative ML model to generate an image containing content corresponding to the selected content adapter and the style adapter is combinable with the generative ML model to adapt the generative ML model to generate an image in a style corresponding to the selected content adapter;predicting, using the hypernetwork, merging coefficients for the selected pair of adapters;merging, using the predicted merging coefficients the selected pair of adapters to form a merged adapter;combining the merged adapter with the generative ML model;generating, using the combined generative ML model and merged adapter, an output image comprising content corresponding to the selected content adapter in the style of the selected style adapter; andupdating the parameters of the hypernetwork to minimize a merging loss for the generated image.

19. A computer-readable storage medium comprising instructions which, when executed by 5 a processor, causes the processor to carry out the method of claims 1 to 16.26 02 26AMENDMENTS TO THE CLAIMS HAVE BEEN FILED AS FOLLOWS39CLAIMS1. A computer-implemented method for real-time generation of a personalized image using a generative machine learning, ML, model, the method comprising:receiving, from a user, content input indicating content to be included in the generated image;receiving, from a user, style input indicating a style for the generated image;obtaining a first low-rank adaptation adapter (LoRA) having a weight update matrix, wherein the first obtained LoRA is a content adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image containing the indicated content;obtaining a second low-rank adaptation adapter (LoRA) having a weight update matrix, wherein the second obtained LoRA is a style adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image in the indicated style;predicting, using a hypernetwork, merging coefficients for merging the obtained content adapter and the obtained style adapter;merging, using the predicted merging coefficients, the obtained content adapter and the obtained style adapter to form a merged adapter;combining the merged adapter with the generative ML model to adapt the generative ML model to generate an image containing the indicated content in the indicated style; andgenerating, using the combined generative ML model and merged adapter, the image comprising the indicated content in the indicated style.

2. The method of claim 1, wherein receiving the content input comprises receiving an image of the content and receiving the style input comprises receiving an image which is in the indicated style.

3. The method of claim 1 or claim 2, wherein predicting merging coefficients comprises predicting merging coefficients by:extracting a column in the weight update matrix for the content adapter;extracting the corresponding column in the weight update matrix for the style adapter;concatenating the extracted columns andrepeating the extracting and concatenating steps for each column in the weight update matrices.

4. The method of claim 3, wherein predicting merging coefficients further comprises: inputting the concatenated columns into the hypernetwork; and26 02 26outputting a plurality of merging coefficients which comprise a merging coefficient for each column of each content adapter and a merging coefficient for each column of each style adapter.

5. The method of claim 4, wherein the hypernetwork comprises a plurality of input layers and inputting the concatenated columns into the hypernetwork further comprises:combining the concatenated columns to form a concatenated matrixselecting an input layer in the hypernetwork based on a size of the concatenated matrix; andinputting the concatenated matrix into the selected input layer of the hypernetwork.

6. The method of any one of claims 1 to 5, wherein the weight update matrices comprise query weights, key weights, value weights and output weights, andpredicting merging coefficients comprises:predicting merging coefficients for one or more of the query weights, key weights, value weights and output weights.

7. The method of claim 6, further comprising:determining whether merging coefficient have been predicted for each of the query weights, key weights, value weights and output weight; andwhen it is determined that merging coeffiicents have not been predicted, using averaged merged coefficients.

8. The method of any one of the preceding claims, further comprising:receiving, from a user, a text input indicating both the content and the style for the generated image.

9. The method of claim 8, wherein generating the image, using the combined generative ML model and merged adapter, comprises using a generative ML model in the form of a text-to-image generative ML model to generate the image using the received text input.

10. The method of claim 9, wherein the text-to-image generative ML model is in the form of a diffusion ML model and generating the image, using the combined generative ML model and merged adapter, further comprises:generating, using a text decoder, a text embedding of the text input;inputting, into the diffusion ML model, the generated text embedding together with noise as a latent seed to obtain an initial latent image representation;26 02 26iteratively denoising the initial latent image representation to obtain an output latent image representation; andconverting, using a decoder, the output latent image representation to the generated image.

11. The method of any one of the preceding claims, further comprising, after generating the image:generating a first prompt for assessing whether the content in the generated image matches the indicated content;generating a second prompt for assessing whether the style of the generated image matches the indicated style;inputting the first and second prompts and the generated image into a multimodal large language model, MLLM; andoutputting from the MLLM an evaluation of whether the generated image accurately represents the indicated content in the indicated style.

12. The method of claim 11, wherein outputting an evaluation from the MLLM comprises: outputting, in response to the first prompt, a first assessment of whether the content in the generated image matches the indicated content;outputting, in response to the second prompt, a second assessment of whether the style of the generated image matches the indicated style; andoutputting an evaluation that the generated image accurately represents the indicated content in the indicated style when the first assessment indicates that there is matching content and the second assessment indicates that there is a matching style.

13. The method of any one of the preceding claims, further comprising, before using the generative ML model, training the hypernetwork by:selecting, from a dataset of adapters, a pair of adapters wherein each pair comprises a content adapter and a style adapter;predicting, using the hypernetwork, merging coefficients for the selected pair of adapters; merging, using the predicted merging coefficients the selected pair of adapters to form a merged adapter;combining the merged adapter with the generative ML model;generating, using the combined generative ML model and merged adapter, an image comprising content corresponding to the selected content adapter in the style of the selected style adapter; and26 02 26updating the parameters of the hypernetwork to minimize a merging loss for the generated image.

14. A computer-implemented method for training a hypernetwork to be used with a generative ML model and a pair of low-rank adaptation adapters (LoRAs) for real-time generation of a personalized image, the training comprising:selecting, from a dataset of LoRAs, a pair of LoRAs wherein each pair comprises a content adapter and a style adapter, the content adapter is combinable with the generative ML model to adapt the generative ML model to generate an image containing content corresponding to the selected content adapter and the style adapter is combinable with the generative ML model to adapt the generative ML model to generate an image in a style corresponding to the selected content adapter;predicting, using the hypernetwork, merging coefficients for the selected pair of adapters;merging, using the predicted merging coefficients, the selected pair of adapters to form a merged adapter;combining the merged adapter with the generative ML model;generating, using the combined generative ML model and merged adapter, an output image comprising content corresponding to the selected content adapter in the style of the selected style adapter; andupdating the parameters of the hypernetwork to minimize a merging loss for the generated image.

15. The method of claim 13 or claim 14, further comprising:generating, using the combined generative ML model and content adapter, a first image;generating, using the combined generative ML model and style adapter, a second image; andcalculating a merging loss from a sum of a difference between the generated output image and the first image and a difference between the generated output image and the second image.

16. A user device for real-time generation of a personalized image using a generative machine learning, ML, model, the user device comprising:at least one processor andmemory storing instructions that, when executed by the at least one processor individually or collectively, cause the user device to generate a personalized image by: receiving, from a user, content input indicating content to be included in the generated image;receiving, from a user, style input indicating a style for the generated image;26 02 26obtaining a first low-rank adaptation adapter (LoRA) having a weight update matrix, wherein the first obtained LoRA is a content adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image containing the indicated content;obtaining a second low-rank adaptation adapter (LoRA) having a weight update matrix, wherein the second obtained LoRA is a style adapter which is combinable with the generative ML model to adapt the generative ML model to generate an image in the indicated style;predicting, using a hypernetwork, merging coefficients for merging the obtained content adapter and the obtained style adapter;merging, using the predicted merging coefficients, the obtained content adapter and the obtained style adapter to form a merged adapter;combining the merged adapter with the generative ML model to adapt the generative ML model to generate an image containing the indicated content in the indicated style; andgenerating, using the combined generative ML model and merged adapter, the image comprising the indicated content in the indicated style.

17. A server comprising:at least one processor andmemory storing instructions that, when executed by the at least one processor individually or collectively, cause the server to train a hypernetwork to be used with a generative ML model and a pair of low-rank adaptation adapters (LoRAs) for real-time generation of a personalized image, by:selecting, from a dataset of LoRAs, a pair of LoRAs wherein each pair comprises a content adapter and a style adapter, the content adapter is combinable with the generative ML model to adapt the generative ML model to generate an image containing content corresponding to the selected content adapter and the style adapter is combinable with the generative ML model to adapt the generative ML model to generate an image in a style corresponding to the selected content adapter;predicting, using the hypernetwork, merging coefficients for the selected pair of adapters;merging, using the predicted merging coefficients the selected pair of adapters to form a merged adapter;combining the merged adapter with the generative ML model;generating, using the combined generative ML model and merged adapter, an output image comprising content corresponding to the selected content adapter in the style of the selected style adapter; andupdating the parameters of the hypernetwork to minimize a merging loss for the generated image.

18. A computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out the method of claims 1 to 15.26 02 26

Images