Image generation method and device and electronic equipment

Abstract

The invention provides an image generation method and device and electronic equipment, and the method comprises the steps: determining a parameter matrix corresponding to specified image information; wherein the specified image information comprises specified image content and / or specified image style; loading the parameter matrix to a pre-trained image generation model, and adjusting specified model parameters in the image generation model through the parameter matrix; inputting preset basic information into the image generation model after parameter adjustment, and outputting a target image; wherein the first part of image information of the target image is matched with the specified image information, and the second part of image information of the target image is matched with the basic information; the basic information comprises a text prompt and / or a basic image. According to the mode, the image migration effect is improved, overfitting is avoided, and accurate control over detailed information and overall information can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of game technology, and in particular to an image generation method, device and electronic equipment. Background Art

[0002] Image transfer can transfer part of the image information from one image to a second image while retaining other parts of the second image information. This image information can include image style and content information. Image transfer can be achieved using pre-trained text-to-image generation models, such as the Stable Diffusion model. However, such models are prone to overfitting when performing image transfer. This means that the model over-focuses on the image's details and ignores the overall information, which affects the image transfer effect. Summary of the Invention

[0003] In view of this, the object of the present invention is to provide an image generation method, device and electronic device to improve the image migration effect, avoid overfitting, and achieve precise control of detail information and overall information.

[0004] In a first aspect, an embodiment of the present invention provides an image generation method, the method comprising: determining a parameter matrix corresponding to specified image information; wherein the specified image information comprises: specified image content and / or specified image style; loading the parameter matrix into a pre-trained image generation model, and adjusting specified model parameters in the image generation model through the parameter matrix; inputting preset basic information into the parameter-adjusted image generation model, and outputting a target image; wherein a first part of the image information of the target image matches the specified image information, and a second part of the image information of the target image matches the basic information; the basic information comprises: text prompts and / or a basic image.

[0005] In a second aspect, an embodiment of the present invention provides an image generation device, which includes: a parameter matrix determination module for determining a parameter matrix corresponding to specified image information; wherein the specified image information includes: specified image content and / or specified image style; a model parameter adjustment module for loading the parameter matrix into a pre-trained image generation model, and adjusting the specified model parameters in the image generation model through the parameter matrix; a target image output module for inputting preset basic information into the image generation model after parameter adjustment, and outputting a target image; wherein the first part of the image information of the target image matches the specified image information, and the second part of the image information of the target image matches the basic information; the basic information includes: text prompts and / or basic images.

[0006] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-mentioned image generation method.

[0007] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned image generation method.

[0008] The embodiments of the present invention bring the following beneficial effects:

[0009] The above-mentioned image generation method, device and electronic device determine the parameter matrix corresponding to the specified image information; wherein the specified image information includes: specified image content and / or specified image style; the parameter matrix is ​​loaded into a pre-trained image generation model, and the specified model parameters in the image generation model are adjusted through the parameter matrix; the preset basic information is input into the image generation model after parameter adjustment, and the target image is output; wherein the first part of the image information of the target image matches the specified image information, and the second part of the image information of the target image matches the basic information; the basic information includes: text prompts and / or basic images.

[0010] In this method, the specified model parameters in the image generation model are adjusted based on the parameter matrix corresponding to the specified image information, and the target image is output through the image generation model after parameter adjustment. This method improves the image migration effect, avoids overfitting, and can achieve precise control of detail information and overall information.

[0011] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0012] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1A flowchart of an image generation method provided by an embodiment of the present invention;

[0015] Figure 2 A schematic diagram of generating a target image based on style and content provided by an embodiment of the present invention;

[0016] Figure 3 A schematic structural diagram of an image generating device provided by an embodiment of the present invention;

[0017] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0019] Currently, through image migration, part of the image information in the first image can be migrated to the second image, while retaining another part of the image information in the second image. This image information may include image style information, image content information, etc.; image migration is usually used in artistic creation, image editing, entertainment and other fields.

[0020] Related technologies can be used to transfer images based on neural networks, such as convolutional neural networks (CNNs) and generative adversarial networks (GANs). However, these methods generally require a lot of computing resources and training time, and it is difficult to precisely control the style and content. Image transfer mainly includes the following methods:

[0021] First, there are image-based iteration methods. These methods use an iterative optimization process to gradually adjust the pixel values ​​of the content image to match the target style distribution. However, this method typically requires a long computation time and is prone to artifacts.

[0022] Second, there's the model-based iteration approach, which achieves style transfer by training a feedforward neural network. However, this approach typically requires a large number of style images for training, and the trained model struggles to adapt to new styles.

[0023] Third, there are approaches based on image decomposition. This type of approach decomposes an image into two independent components: content and style, and then processes these two components separately. However, since the content and style of an image are often highly coupled, this decomposition is often difficult to implement.

[0024] Another approach is to achieve image transfer using a pre-trained text-to-image generative model, such as the StableDiffusion model. However, this model is prone to overfitting when performing image transfer. Overfitting typically occurs when the model is too complex, the training data is too limited, or the training time is too long. Although the model performs well on the training data, it performs poorly on the actual application data. In other words, the model over-focuses on the image's details and ignores the overall information, which affects the image transfer effect.

[0025] Based on this, an image generation method, apparatus, and electronic device provided by embodiments of the present invention can be applied to image migration, for example, migrating image style information or image content information from one image to another.

[0026] To facilitate understanding of this embodiment, an image generation method disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the image generation method includes the following steps:

[0027] Step S102, determining a parameter matrix corresponding to the specified image information; wherein the specified image information includes: specified image content and / or specified image style;

[0028] The designated image information may be designated image content or designated image style, or may include both designated image content and designated image style.

[0029] The parameter matrix corresponds to the specified image information. For example, when the specified image information is the specified image content, the parameter matrix corresponds to the specified image content; when the specified image information is the specified image style, the parameter matrix corresponds to the specified image style.

[0030] The parameter matrix can be pre-trained and typically differs when the specified image information differs. In one example, the parameter matrix is ​​a learnable weight matrix, specifically a LoRA (Low-Rank Adaptation) matrix. During training, the loss value can be used to measure the difference between the global or local features of the model output image and the input image, thereby fine-tuning the parameter matrix.

[0031] Optionally, the specified image information includes specified image content and specified image style, and then two parameter matrices are set to correspond to the specified image content and the specified image style respectively.

[0032] Step S104, loading the parameter matrix into the pre-trained image generation model, and adjusting the specified model parameters in the image generation model through the parameter matrix;

[0033] The image generation model usually includes multiple modules. According to the application scenario and actual needs, modules can be selected from the multiple modules. For example, the image generation model includes four modules, then the third module can be set as a style module to encode the style information of the image, and the fourth module can be set as a content module to encode the content information of the image.

[0034] The specified model parameters generally include module parameters in different modules of the image generation model; for example, the specified model parameters are module parameters in different modules at different levels in the image generation model. The modules used to encode image information can be subdivided, such as style information can be subdivided into texture information, brushstroke information, and color information, and content information can be subdivided into shape information, structure information, and layout information.

[0035] For example, when the image generation model is a Stable Diffusion model, the model includes multiple Transformer blocks. You can select some Transformer blocks and adjust the model parameters in these Transformer blocks through the above parameter matrix; for example, CrossAttnMidBlock block, CrossAttnDownBlock_10_2 block, CrossAttnUpBlock_10_3 block, etc.

[0036] The Transformer block includes multiple operation layers, such as self-attention layer, cross-attention layer, linear operation layer, etc.; the above-mentioned specified model parameters can be all or part of the operation parameters of the specified operation layer in the selected Transformer block. For example, the specified model parameters include all the operation parameters of the self-attention layer and the cross-attention layer in the selected Transformer block.

[0037] By loading the parameter matrices corresponding to different specified image information into a pre-trained image generation model, the specified model parameters in the image generation model can be adjusted. For example, a low-rank parameter matrix can be superimposed on the original matrix parameters of the pre-trained image generation model to obtain the adjusted specified model parameters.

[0038] In one embodiment, the parameter matrix can be loaded into a pre-trained image generation model, which generally includes different layers, specifically a downsampling path, an upsampling path or an intermediate layer, and Transformer blocks are arranged in different layers, that is, a neural network structure based on a self-attention mechanism and a multi-head attention mechanism.

[0039] In practical applications, two Transformer blocks at different levels can be selected in the image generation model, and then the parameter matrix corresponding to the specified image information can be loaded into the corresponding Transformer block, which can be the self-attention layer or the cross-attention layer in the Transformer block. The weight matrices in the different attention layers are the original matrix parameters that have been pre-trained. The module parameters in the content module can be adjusted by specifying the parameter matrix corresponding to the image content, or the module parameters in the style module can be adjusted by specifying the parameter matrix corresponding to the image style. The module parameters in the corresponding module can be further adjusted by the parameter matrix corresponding to the shape information and structure information, and the module parameters in the corresponding module can be adjusted by the parameter matrix corresponding to the texture information and stroke information. In this embodiment, there is no limitation on the specified model parameters and the corresponding parameter matrix.

[0040] In one example, the pre-trained image generation model may be a Stable Diffusion XL (SDXL) model, a Flux model, or the like.

[0041] Step S106, input the preset basic information into the image generation model after parameter adjustment, and output the target image; wherein the first part of the image information of the target image matches the specified image information, and the second part of the image information of the target image matches the basic information; the basic information includes: text prompts and / or basic images.

[0042] The preset basic information can be a text prompt or a basic image, or both. The text prompt can be a word, a phrase, or a sentence, such as an object, a scene, a layout, or an oil painting style or watercolor style. The first portion of image information is typically a capability acquired by the image generation model after training. The basic image can be a single image, including image style and image content, and is used to guide the second portion of image information of the target image generated by the image generation model to match the basic information.

[0043] In actual implementation, if only style or content needs to be applied, and the first part of the image information of the target image matches the specified image style, or the specified image content, then the text prompt can be input into the image generation model after parameter adjustment, and the image generation model outputs the target image, and the first part of the image information of the target image matches the specified image style, or the specified image content, and the second part of the image information of the target image matches the text prompt.

[0044] In one embodiment, if it is necessary to apply style or content at the same time, only text prompts can be input into the image generation model after parameter adjustment. Through text prompts, more specific and flexible control can be performed to generate elements in the target image that are not included in the training image, or to provide a more detailed description of style and content.

[0045] The above-mentioned image generation method determines the parameter matrix corresponding to the specified image information; wherein the specified image information includes: specified image content and / or specified image style; the parameter matrix is ​​loaded into a pre-trained image generation model, and the specified model parameters in the image generation model are adjusted through the parameter matrix; the preset basic information is input into the image generation model after parameter adjustment, and the target image is output; wherein the first part of the image information of the target image matches the specified image information, and the second part of the image information of the target image matches the basic information; the basic information includes: text prompts and / or basic images.

[0046] In this method, the specified model parameters in the image generation model are adjusted based on the parameter matrix corresponding to the specified image information, and the target image is output through the image generation model after parameter adjustment. This method improves the image migration effect, avoids overfitting, and can achieve precise control of detail information and overall information.

[0047] In a specific implementation, the parameter matrix is ​​preset with a matrix index, and the matrix index is used to indicate that the parameter matrix is ​​loaded into a specified model position in the image generation model.

[0048] The parameter matrix is ​​pre-set with a matrix index, which can be understood as a 'position identifier' of the parameter matrix in the image generation model.

[0049] The designated model position can be strategically and purposefully selected based on the application scenario. For example, the position and function of different modules in the image generation model can be comprehensively considered, and then the designated model position, such as the module or the attention layer in the module, can be selected.

[0050] Load the parameter matrix into the specified model position indicated by the matrix index in the pre-trained image generation model, and adjust the specified model parameters in the specified model position through the parameter matrix.

[0051] In actual implementation, the parameter matrix is ​​loaded into the pre-trained image generation model according to the specified model position indicated by the matrix index preset in the parameter matrix. For example, the matrix index indicates the self-attention layer of the fourth module in the image generation model, or indicates the cross-attention layer of the fifth module in the image generation model.

[0052] After completing the loading of the parameter matrix, adjust the specified model parameters in the specified model position according to the parameter matrix corresponding to the encoding task information. For example, based on the above example, after loading the parameter matrix into the self-attention layer of the fourth module, adjust the parameters of the style information through the parameter matrix, or, after loading the parameter matrix into the cross-attention layer of the fifth module, adjust the parameters of the content information through the parameter matrix.

[0053] Furthermore, the matrix index is specifically used to indicate that the parameter matrix is ​​loaded into a specified feature conversion module in the image generation model.

[0054] The specified feature conversion module is included in the image generation model, for example, the SDXL_SpatialTransformer_X module in the SDXL model; the specified feature conversion module can be understood as a subnetwork, which consists of multiple layers and components. For example, the layers include self-attention layers and cross-attention layers, and the components include feedforward networks, layer normalization, and residual connection networks; the specified feature conversion module usually has a one-to-one correspondence with the parameter matrix to avoid the many-to-one situation where the use of multiple specified feature conversion modules increases the complexity of model training.

[0055] In practical applications, the number of specified feature conversion modules is at least two, and the specific number can be adjusted according to the application scenario and training results.

[0056] The parameter matrix is ​​loaded into the specified feature conversion module indicated by the matrix index in the pre-trained image generation model, and the module parameters in the specified feature conversion module are adjusted through the parameter matrix.

[0057] In actual implementation, the parameter matrix is ​​loaded into the pre-trained image generation model according to the specified feature conversion module indicated by the matrix index preset in the parameter matrix. For example, the matrix index includes the style block index style_block_index and the content block index content_block_index. Assuming that the value of the style block index is 2, it indicates that the parameter matrix for encoding style information is loaded into the second module in the pre-image generation model; assuming that the value of the content block index is 3, it indicates that the parameter matrix for encoding content information is loaded into the third module in the image generation model.

[0058] After completing the loading of the parameter matrix, the module parameters in the specified feature conversion module are adjusted through the encoding information corresponding to the parameter matrix. For example, based on the above example, after loading the parameter matrix into the second module, the parameters of the style information are adjusted through the parameter matrix, or, after loading the parameter matrix into the third module, the parameters of the content information are adjusted through the parameter matrix.

[0059] It should be noted that selecting a specific feature conversion module is a process that requires experimentation and tuning, and there is no limitation on the specific feature conversion module here.

[0060] Furthermore, the feature conversion module is specified to include: a self-attention layer and a cross-attention layer.

[0061] This self-attention layer enables the model to focus on the relationship between different positions in the input sequence, guiding the model to understand the local features and overall structure of the image. This is important for capturing both the style and content information of the image. For example, if the input sequence is a feature map of a cat, the self-attention layer can guide the model to recognize the relationship between "cat's eyes" and "cat's nose," thereby generating more realistic cat images.

[0062] The cross-attention layer enables the model to focus on the relationship between two different input sequences. It is used to process the relationship between image features and text features, which is important for capturing both the style and content information of the image. For example, if the input sequence is a feature map of a cat and a text description of "a cat wearing a space suit", the cross-attention layer can guide the model to understand this text description and add the element of "space suit" to the image of "cat".

[0063] Adjust the self-attention layer and / or cross-attention layer in the specified feature transformation module through the parameter matrix.

[0064] In actual implementation, the self-attention layer or the cross-attention layer in the specified feature conversion module can be adjusted through the parameter matrix; the self-attention layer and the cross-attention layer in the specified feature conversion module can also be adjusted simultaneously through the parameter matrix.

[0065] In one example, the pre-set designated feature conversion modules are the fourth and fifth modules in the image generation model. After loading the parameter matrix into them, the parameter matrix is ​​used to adjust the self-attention layer and cross-attention layer of the module respectively. By using the parameter matrix in the two attention layers, the model's attention to the information in the module, such as style information and content information, can be more flexibly adjusted.

[0066] In an optional method, when the feature conversion module is specified to correspond to the fourth module in the image generation model, the fourth module is used as a style module for encoding the style information of the image; when it corresponds to the fifth module in the image generation model, the fifth module is used as a content module for encoding the content information of the image. Then, by adjusting the parameters of the self-attention layer and the cross-attention layer in the style module through the parameter matrix, the model can pay more attention to the style features of the image, such as texture, color, brushstrokes, etc.; by adjusting the parameters of the self-attention layer and the cross-attention layer in the content module through the parameter matrix, the model can pay more attention to the content features of the image, such as the shape, structure, layout, etc. of the object.

[0067] The above method can make image migration more flexible by adjusting the self-attention layer and cross-attention layer in the specified feature conversion module.

[0068] In one implementation, the designated image information includes multiple types; different designated image information corresponds to different designated feature conversion modules set by parameter matrices.

[0069] The designated image information may include multiple types, such as sub-styles divided according to style, such as color style, texture style, and brushstroke style, and sub-contents divided according to content, such as object shape, object position, and object posture.

[0070] In practical applications, if the style or content of the generated image needs to be more refined, the style can be divided into multiple sub-styles, or the content can be divided into multiple sub-contents. For example, the specified image information includes color style, texture style, and object shape. Then in the image generation model, the parameter matrices corresponding to the three specified image information have different specified feature conversion modules.

[0071] It can be understood that if more image information is specified, the more parameter matrices corresponding to the specified image information will be specified, that is, the more feature conversion modules will be specified. The number of specified feature conversion modules and the specified image information can be adjusted according to the specific image migration task and experimental results.

[0072] In one implementation, specifying image information includes specifying image content and specifying image style;

[0073] In one example, a Transformer module can be set in the image generation model for a specified image content and a specified image style respectively.

[0074] The specified feature conversion module of the parameter matrix setting corresponding to the specified image content includes: the intermediate-level feature conversion module in the U-Net network of the image generation model; the specified feature conversion module of the parameter matrix setting corresponding to the specified image style includes: the feature conversion module in the downsampling path or the upsampling path in the U-Net network of the image generation model.

[0075] The U-Net network includes an intermediate-level feature conversion module, a downsampling path, and an upsampling path. The downsampling path is used to capture the contextual information of the image and process the local and detailed features of the image. The upsampling path is used to restore the details and spatial resolution of the image. The intermediate-level feature conversion module is used to capture the overall structure and content of the image and process the global and semantic features of the image, thereby making the generated image clearer and more realistic.

[0076] In actual implementation, two independent Transformer modules can be set, one of which is a specified feature conversion module set as a parameter matrix corresponding to the specified image content. Preferably, the intermediate-level feature conversion module in the U-Net network is used to extract or migrate the content information of the image through the intermediate-level feature conversion module. For example, the content information is the shape, structure, layout, etc. of the object.

[0077] Another specified feature conversion module is set as a parameter matrix corresponding to the specified image style. Preferably, the feature conversion module in the downsampling path or the upsampling path in the U-Net network is used. By using the feature conversion module in the downsampling path, the style information of the image can be extracted or migrated. For example, the style information is texture, brushstroke, color and other styles. By using the feature conversion module in the upsampling path, certain detail features can be retained or enhanced when generating the image, or the style can be more finely controlled.

[0078] In one implementation, the parameter matrix is ​​obtained through pre-training, and the parameter matrix includes: an information capture sub-matrix and an information expression sub-matrix.

[0079] That is, the parameter matrix can be obtained by pre-training the initial matrix, and the information capture submatrix and the information expression submatrix contained in the parameter matrix have their own dimensions, initialization values ​​and training parameters respectively.

[0080] This information-capturing submatrix is ​​used to capture the style or content characteristics of an image. It can map high-dimensional image features into a low-dimensional "style space" or "content space." For example, the information-capturing submatrix is ​​matrix A, which can reduce the dimensionality of the input matrix from d dimensions to r dimensions. Where d is greater than r, the information-capturing submatrix can be considered a "feature extractor." Optionally, if the image information includes both content and style information, separate information-capturing submatrices are set for the content and style information.

[0081] The information expression submatrix is ​​used to map features in the "style space or content space" to the image feature space, thereby affecting the image effect of the model output. For example, the information expression submatrix is ​​matrix B, which can upgrade the input matrix from r dimensions to k dimensions. r is less than k, and the information expression submatrix can be understood as a "feature decoder."

[0082] Determine a matrix product of an information capture submatrix and an information expression submatrix, wherein the dimensions of the matrix product match the dimensions of the original parameters of the specified model parameters; and superimpose the matrix product on the original parameters of the specified model parameters to obtain the specified model parameters in the adjusted image generation model.

[0083] In actual implementation, the matrix product of the information capture submatrix and the information expression submatrix is ​​first calculated. For example, based on the above example, the matrix product is BA, and the dimension of the matrix product matches the dimension of the original parameter of the specified model parameter. Assuming that the original parameter of the specified model parameter is W0 and the dimension of the original parameter is three-dimensional, the dimension of the matrix product is also three-dimensional.

[0084] After obtaining the matrix product, the matrix product is superimposed on the original parameters of the specified model parameters to obtain the specified model parameters W in the adjusted image generation model. The specific formula is as follows:

[0085] W=W0+ΔW=W0+BA.

[0086] In an example, the parameter matrix is ​​a LoRA matrix. The rank can control the amount of information in the LoRA matrix. In the information capture submatrix A and the information expression submatrix B, the larger the rank, the stronger the expressive ability of the LoRA matrix, but it is also more prone to overfitting; the smaller the rank, the weaker the expressive ability of the LoRA matrix, but it is less likely to overfit.

[0087] The original parameters of the above-mentioned specified model parameters can be the weight matrix in a self-attention layer or cross-attention layer in the image generation model, for example, the query matrix (Query, Q), the key matrix (Key, K) and the value matrix (Value, V). Since the weight matrix is ​​trained in the pre-trained image generation model, the number of training parameters can be greatly reduced, thereby improving training efficiency.

[0088] In one approach, the parameter matrix corresponding to the specified image information is trained as follows:

[0089] An initial matrix is ​​loaded into a specified feature conversion module in an image generation model; wherein the initial matrix is ​​used to adjust module parameters of the specified feature conversion module; a preset reference image and preset training prompt information are input into the image generation model to obtain an output image; wherein the reference image contains specified image information; a loss value is determined based on the output image and the reference image; matrix parameters of the initial matrix are adjusted based on the loss value until the first part of the image information of the output image matches the specified image information, and the adjusted initial matrix is ​​determined as a parameter matrix.

[0090] The matrix parameters of the initial matrix can usually be adjusted through training. The initial matrix can specifically include parameters such as a first initial sub-matrix, a second initial sub-matrix, and a scaling factor for adjusting the influence of the matrix.

[0091] The preset reference image may include style information and content information, which may be represented using features at different levels, specifically including low-level features, mid-level features, and high-level features. The preset training prompt information may be a general text prompt, i.e., the training prompt information does not include any specific information about the image style or content, so that the designated feature conversion module learns features of the reference image, such as the style information and content information of the reference image.

[0092] In actual implementation, the initial matrix is ​​loaded into a specified feature conversion module in the image generation model, such as a module for encoding style information and a module for encoding content information, and the module parameters of the specified feature conversion module are adjusted through the initial matrix.

[0093] In one example, during the training phase, two independent LoRA modules are constructed, including a style LoRA module and a content LoRA module. Each LoRA module is associated with a Transformer block in the pre-trained image generation model. For example, the style LoRA module may be associated with the fourth Transformer module, and the content LoRA module may be associated with the fifth Transformer module. The module parameters in the style LoRA module can be adjusted using the initial matrix corresponding to the fourth Transformer module, and the module parameters in the content LoRA module can be adjusted using the initial matrix corresponding to the fifth Transformer module.

[0094] During training, a preset reference image and preset training prompt information are input into the image generation model. The reference image can be a single image, and the training prompt information can be a general text prompt, such as 'A[v]'. In other words, inputting a single image containing style and content information and a general text prompt into the image generation model allows different designated feature conversion modules to learn different aspects of the single image. For example, the style LoRA module extracts style information from a single image, and the content LoRA module extracts content information from a single image, thereby decoupling style and content.

[0095] After obtaining the output image, the loss value can be calculated based on the output image and the reference image. This loss value can be used to measure the difference between the model's predicted results and the actual results, thereby reflecting the model's true goal and guiding the specified feature conversion module to learn the underlying laws of the data.

[0096] After determining the loss value, the optimizer can update the matrix parameters of the initial matrix according to the loss value. The optimizer, such as AdamW, can then update the matrix parameters of the initial matrix at least once until the first part of the image information of the output image matches the specified image information, for example, the style information of the output image matches the specified image style, or the content information of the output image matches the specified image content; and then the adjusted initial matrix is ​​used as the final parameter matrix.

[0097] Optionally, the rank of the initial matrix can be limited during training to reduce the complexity of the model; or, early stopping can be used, that is, training can be stopped when the model starts to overfit the training data; or, data augmentation can be performed on the reference image, such as random cropping, rotation, flipping, etc., to increase the diversity of the training data and reduce the risk of model overfitting.

[0098] In one implementation, the reference image is a single image.

[0099] In practice, a single image can contain both style and content information. The single image input to the image generation model can be preprocessed, including resizing and cropping, to ensure that it meets the preset standards of the pre-trained image generation model. For example, the single image can be resized to the resolution of the pre-trained image generation model, then the resized image can be center-cropped to obtain a square image. The square image can then be normalized to obtain the final image.

[0100] In one implementation, the initial matrix includes: a first initial submatrix and a second initial submatrix; wherein the first initial submatrix corresponds to the information capture submatrix in the parameter matrix; the second initial submatrix corresponds to the information expression submatrix in the parameter matrix; the initial values ​​of the parameters of the first initial submatrix are Gaussian distribution, and the initial values ​​of the parameters of the second initial submatrix are zero.

[0101] The first initial sub-matrix may correspond to the information capturing sub-matrix in the parameter matrix. Randomly initializing the first initial sub-matrix may enable the first initial sub-matrix to have the ability to capture information.

[0102] For example, the initial values ​​of the parameters of the first initial submatrix are Gaussian distribution, which is also called normal distribution. In practical applications, Gaussian distribution can indicate that the elements in the first initial submatrix A have a mean of zero and a standard deviation randomly drawn from a normal distribution with a smaller value.

[0103] The second initial submatrix corresponds to the information expression submatrix in the parameter matrix. Initializing the first initial submatrix to zero can ensure that the original output of the model is not affected at the beginning of training.

[0104] That is to say, the initial value of the parameter of the second initial submatrix is ​​zero, that is, initializing the second initial submatrix B to an all-zero matrix will not affect the output of the pre-trained image generation model.

[0105] In one implementation, a preset reference image and preset training prompt information are input into an image generation model. Through the image generation model, the reference image is encoded into a latent feature vector, and the training prompt information is converted into an embedded feature vector. A random noise vector is generated that matches the vector dimension of the latent feature vector. According to a preset time step, the random noise vector is added to the latent feature vector. Based on the latent feature vector with the added random noise vector and the embedded feature vector, an output image is obtained.

[0106] The preset time step may refer to the number of consecutive observation points input into the model, and the time step may be randomly generated. The embedded feature vector may interact with the image features through a cross-attention layer, thereby affecting the image generation process.

[0107] During the training process, a preset reference image is input into the image generation model, and then the input reference image can be encoded into a latent feature vector by the encoder of the VAE variational autoencoder in the image generation model. At the same time, the preset training prompt information can be converted into an embedded feature vector by the text encoder in the image generation model. For example, the general text prompt 'A[v]' is encoded into an embedded feature vector. The text encoder can be CLIP (Contrastive Language-Image Pre-Training).

[0108] Furthermore, the scheduler of the image generation model adds noise to the latent feature vector to generate a random noise vector that matches the vector dimension of the latent feature vector, and adds the random noise vector to the latent feature vector according to a preset time step; then the embedded feature vector and the latent feature vector with the random noise vector added can be received by the U-Net network of the image generation model, and the noise is predicted to obtain an output image. Optionally, the style LoRA module predicts the noise based on the latent feature vector, time step and embedded feature vector of the added noise vector, or the content LoRA module predicts the noise based on the latent feature vector, time step and embedded feature vector of the added noise vector, and thus the output image is obtained.

[0109] In one implementation, a first loss value is obtained based on the global features of the output image and the reference image; wherein the first loss value is used to indicate whether the output image contains the global features of the reference image; a second loss value is obtained based on the specified information features of the output image and the reference image; wherein the second loss value is used to indicate whether the output image contains the specified information features of the reference image; and a final loss value is determined based on the first loss value and the second loss value.

[0110] The designated information feature may be a content feature, or a style feature, or a sub-content feature of a content feature, or a sub-style feature of a style feature.

[0111] In actual implementation, a first loss value is calculated based on the global features of the output image and the reference image. This first loss value indicates whether the output image contains the global features of the reference image. The first loss value can be used to measure the difference between the global features of the output image and the reference image, thereby adjusting the matrix parameters and guiding the model to learn the global features of the reference image. Specifically, the first loss value can be calculated using the mean square error (MSE) or other similar loss functions.

[0112] In one example, a second loss value is calculated based on the specified information features of the output image and the reference image. This second loss value indicates whether the output image contains the specified information features of the reference image. The second loss value can be used to measure the difference between the specified information features of the output image and the reference image, thereby adjusting the matrix parameters to guide the model to learn the specified information features of the reference image. Specifically, the second loss value can be calculated using the mean square error (MSE) or other similar loss functions.

[0113] In the above method, the loss value includes the first loss value and the second loss value, which is conducive to adjusting the matrix parameters during training, thereby avoiding overfitting of the model style or overfitting of the content.

[0114] In one implementation, based on a preset weight parameter, a weighted sum of the first loss value and the second loss value is determined as the final loss value.

[0115] In actual implementation, the first loss value corresponds to a preset weight parameter λ1, and the second loss value corresponds to a preset weight parameter λ2; the product of the first loss value L1 and the preset weight parameter of the first loss value is calculated, and the product of the second loss value L2 and the preset weight parameter L2 of the second loss value is calculated, and the two are added together to obtain the final loss value, as shown in the formula: L=λ1L1+λ2L2.

[0116] In the above method, by setting the preset weight parameters, the importance of different loss values ​​can be balanced.

[0117] In one implementation, specifying image information includes specifying image content and specifying image style.

[0118] Based on the specified image content of the output image and the reference image, a first sub-loss value is obtained; wherein the first sub-loss value is used to indicate whether the output image contains the specified image content of the reference image; based on the specified image style of the output image and the reference image, a second sub-loss value is obtained; wherein the second sub-loss value is used to indicate whether the output image contains the specified image style of the reference image; based on the first sub-loss value and the second sub-loss value, a second loss value is determined.

[0119] In actual implementation, the first sub-loss value Lstyle is calculated based on the specified image content of the output image and the reference image. The first sub-loss value indicates whether the output image contains the specified image content of the reference image. The first sub-loss value can be used to measure the difference between the specified image content of the output image and the reference image, and then adjust the matrix parameters to guide the model to learn the specified image content of the reference image.

[0120] In one example, based on the specified image styles of the output image and the reference image, a second sub-loss value Lcontent is calculated. The second sub-loss value indicates whether the output image contains the specified image style of the reference image. The second sub-loss value can be used to measure the difference between the specified image styles of the output image and the reference image, and then the matrix parameters can be adjusted to guide the model to learn the specified image style of the reference image.

[0121] After obtaining the first sub-loss value and the second sub-loss value, the sum of the first sub-loss value and the second sub-loss value can be determined as the second loss value.

[0122] In one embodiment, Figure 2 As shown, in (a), (b), and (c), the specified image style is such as Style in a row, the specified image style is such as Input Content in a column, and the target image is such as the image corresponding to the row and column.

[0123] The embodiments of the present application have the following advantages:

[0124] In this embodiment, an improved dual-branch LoRA training strategy can be implemented to quickly and effectively extract style and content information from a single image and encode them into two independent LoRA modules respectively; these two modules can be flexibly combined to achieve fine control of the style and content of the generated image, thereby generating high-quality style transfer images.

[0125] This embodiment only requires a single image as input to achieve high-quality style transfer, without requiring a large amount of training data or a complex optimization process.

[0126] The training strategy based on LoRA in this embodiment has high computational efficiency and can complete model training in a short time.

[0127] The LoRA modules trained in this embodiment can be flexibly combined and applied to different content images and text prompts to generate diverse style transfer results.

[0128] This embodiment effectively avoids the overfitting problem common in traditional LoRA methods through special loss function design and training process.

[0129] This embodiment can be applied to artistic creation. Artists can use this method to quickly convert their photos or paintings into works with a specific artistic style. In image editing, users can use this method to easily change the style of photos, such as converting photos into oil paintings, watercolors, or sketches. In entertainment, for example, it can be applied to games, social media and other fields to provide users with a personalized image stylization experience. In design, it provides designers with a fast style exploration tool to help them quickly generate design solutions with different styles.

[0130] The embodiment of the present invention also provides a schematic diagram of an image processing device, such as Figure 3 As shown, the device includes:

[0131] The parameter matrix determination module 31 is used to determine the parameter matrix corresponding to the specified image information; wherein the specified image information includes: specified image content and / or specified image style;

[0132] A model parameter adjustment module 32 is used to load the parameter matrix into the pre-trained image generation model and adjust the specified model parameters in the image generation model through the parameter matrix;

[0133] The target image output module 33 is used to input the preset basic information into the image generation model after parameter adjustment and output the target image; wherein the first part of the image information of the target image matches the specified image information, and the second part of the image information of the target image matches the basic information; the basic information includes: text prompts and / or basic images.

[0134] The above-mentioned image generation device determines a parameter matrix corresponding to specified image information; wherein the specified image information includes: specified image content and / or specified image style; loads the parameter matrix into a pre-trained image generation model, and adjusts the specified model parameters in the image generation model through the parameter matrix; inputs preset basic information into the image generation model after parameter adjustment, and outputs a target image; wherein the first part of the image information of the target image matches the specified image information, and the second part of the image information of the target image matches the basic information; the basic information includes: text prompts and / or basic images.

[0135] In this method, the specified model parameters in the image generation model are adjusted based on the parameter matrix corresponding to the specified image information, and the target image is output through the image generation model after parameter adjustment. This method improves the image migration effect, avoids overfitting, and can achieve precise control of detail information and overall information.

[0136] The above-mentioned parameter matrix is ​​preset with a matrix index, and the matrix index is used to: indicate that the parameter matrix is ​​loaded into the specified model position in the image generation model; the above-mentioned model parameter adjustment module is also used to: load the parameter matrix into the specified model position indicated by the matrix index in the pre-trained image generation model, and adjust the specified model parameters in the specified model position through the parameter matrix.

[0137] The above-mentioned matrix index is specifically used to: indicate that the parameter matrix is ​​loaded into the specified feature conversion module in the image generation model; the above-mentioned model parameter adjustment module is also used to: load the parameter matrix into the specified feature conversion module indicated by the matrix index in the pre-trained image generation model, and adjust the module parameters in the specified feature conversion module through the parameter matrix.

[0138] The above-mentioned specified feature conversion module includes: a self-attention layer and a cross-attention layer; the above-mentioned model parameter adjustment module is also used to: adjust the self-attention layer and / or cross-attention layer in the specified feature conversion module through the parameter matrix.

[0139] The above-mentioned designated image information includes multiple types; different designated image information corresponds to different designated feature conversion modules set by parameter matrices.

[0140] The above-mentioned specified image information includes specified image content and specified image style; the specified feature conversion module set by the parameter matrix corresponding to the above-mentioned specified image content includes: an intermediate-level feature conversion module in the U-Net network of the image generation model; the specified feature conversion module set by the parameter matrix corresponding to the specified image style includes: a feature conversion module in the downsampling path or the upsampling path in the U-Net network of the image generation model.

[0141] The above-mentioned parameter matrix is ​​obtained through pre-training, and the parameter matrix includes: an information capture sub-matrix and an information expression sub-matrix; the above-mentioned model parameter adjustment module is also used to: determine the matrix product of the information capture sub-matrix and the information expression sub-matrix; wherein the dimension of the matrix product matches the dimension of the original parameters of the specified model parameters; superimpose the matrix product on the original parameters of the specified model parameters to obtain the specified model parameters in the adjusted image generation model.

[0142] The parameter matrix corresponding to the above-mentioned specified image information is trained in the following manner: loading the initial matrix into the specified feature conversion module in the image generation model; wherein the initial matrix is ​​used to adjust the module parameters of the specified feature conversion module; inputting a preset reference image and preset training prompt information into the image generation model to obtain an output image; wherein the reference image contains the specified image information; determining a loss value based on the output image and the reference image; adjusting the matrix parameters of the initial matrix based on the loss value until the first part of the image information of the output image matches the specified image information, and determining the adjusted initial matrix as the parameter matrix.

[0143] The above reference images are single images.

[0144] The above-mentioned initial matrix includes: a first initial submatrix and a second initial submatrix; wherein, the first initial submatrix corresponds to the information capture submatrix in the parameter matrix; the second initial submatrix corresponds to the information expression submatrix in the parameter matrix; the initial values ​​of the parameters of the first initial submatrix are Gaussian distribution, and the initial values ​​of the parameters of the second initial submatrix are zero.

[0145] The parameter matrix determination module is further configured to: input a preset reference image and preset training prompt information into an image generation model, encode the reference image into a latent feature vector through the image generation model, and convert the training prompt information into an embedded feature vector; generate a random noise vector that matches the vector dimension of the latent feature vector; add the random noise vector to the latent feature vector according to a preset time step; and obtain an output image based on the latent feature vector to which the random noise vector is added and the embedded feature vector.

[0146] The above-mentioned parameter matrix determination module is also used to: obtain a first loss value based on the global features of the output image and the reference image; wherein the first loss value is used to: indicate whether the output image contains the global features of the reference image; obtain a second loss value based on the specified information features of the output image and the reference image; wherein the second loss value is used to: indicate whether the output image contains the specified information features of the reference image; and determine the final loss value based on the first loss value and the second loss value.

[0147] The parameter matrix determination module is further configured to determine, based on a preset weight parameter, a weighted sum of the first loss value and the second loss value as a final loss value.

[0148] The above-mentioned specified image information includes: specified image content and specified image style; the above-mentioned parameter matrix determination module is also used to: obtain a first sub-loss value based on the specified image content of the output image and the reference image; wherein the first sub-loss value is used to: indicate whether the output image contains the specified image content of the reference image; obtain a second sub-loss value based on the specified image style of the output image and the reference image; wherein the second sub-loss value is used to: indicate whether the output image contains the specified image style of the reference image; determine the second loss value based on the first sub-loss value and the second sub-loss value.

[0149] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-mentioned image generation method. The electronic device can be a server or a terminal device.

[0150] See also Figure 4 As shown, the electronic device includes a processor 100 and a memory 101 , wherein the memory 101 stores computer-executable instructions that can be executed by the processor 100 , and the processor 100 executes the computer-executable instructions to implement the above-mentioned image generation method.

[0151] Furthermore, Figure 4 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected via the bus 102 .

[0152] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is achieved through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0153] The processor 100 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 100 or software instructions. The above processor 100 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or register. The storage medium is located in the memory 101. The processor 100 reads the information in the memory 101 and, in conjunction with its hardware, completes the steps of the method of the aforementioned embodiment.

[0154] The processor in the electronic device can implement the following operations in the image generation method by executing computer-executable instructions:

[0155] Determine a parameter matrix corresponding to specified image information; wherein the specified image information includes: specified image content and / or specified image style; load the parameter matrix into a pre-trained image generation model, and adjust specified model parameters in the image generation model through the parameter matrix; input preset basic information into the image generation model after parameter adjustment, and output a target image; wherein a first portion of image information of the target image matches the specified image information, and a second portion of image information of the target image matches the basic information; the basic information includes: text prompts and / or a basic image.

[0156] The parameter matrix is ​​preset with a matrix index, which is used to: indicate that the parameter matrix is ​​loaded into a specified model position in the image generation model; load the parameter matrix into the specified model position indicated by the matrix index in the pre-trained image generation model, and adjust the specified model parameters in the specified model position through the parameter matrix.

[0157] The matrix index is specifically used to: indicate that the parameter matrix is ​​loaded into a specified feature conversion module in the image generation model; load the parameter matrix into the specified feature conversion module indicated by the matrix index in the pre-trained image generation model, and adjust the module parameters in the specified feature conversion module through the parameter matrix.

[0158] The specified feature conversion module includes: a self-attention layer and a cross-attention layer; the self-attention layer and / or the cross-attention layer in the specified feature conversion module is adjusted through a parameter matrix.

[0159] The specified image information includes multiple types; different specified image information corresponds to different specified feature conversion modules with different parameter matrix settings.

[0160] The specified image information includes the specified image content and the specified image style; the intermediate-level feature conversion module in the U-Net network of the image generation model; the specified feature conversion module with the parameter matrix setting corresponding to the specified image style, including: the feature conversion module in the downsampling path or the upsampling path in the U-Net network of the image generation model.

[0161] The parameter matrix is ​​obtained through pre-training, and the parameter matrix includes: an information capture submatrix and an information expression submatrix; determining the matrix product of the information capture submatrix and the information expression submatrix; wherein the dimension of the matrix product matches the dimension of the original parameters of the specified model parameters; superimposing the matrix product on the original parameters of the specified model parameters to obtain the specified model parameters in the adjusted image generation model.

[0162] The parameter matrix corresponding to the specified image information is trained in the following manner: loading an initial matrix into a specified feature conversion module in an image generation model; wherein the initial matrix is ​​used to adjust module parameters of the specified feature conversion module; inputting a preset reference image and preset training prompt information into the image generation model to obtain an output image; wherein the reference image contains the specified image information; determining a loss value based on the output image and the reference image; adjusting matrix parameters of the initial matrix based on the loss value until the first part of the image information of the output image matches the specified image information, and determining the adjusted initial matrix as the parameter matrix.

[0163] The reference image is a single image.

[0164] The initial matrix includes: a first initial submatrix and a second initial submatrix; wherein the first initial submatrix corresponds to the information capture submatrix in the parameter matrix; the second initial submatrix corresponds to the information expression submatrix in the parameter matrix; the initial values ​​of the parameters of the first initial submatrix are Gaussian distribution, and the initial values ​​of the parameters of the second initial submatrix are zero.

[0165] A preset reference image and preset training prompt information are input into an image generation model. Through the image generation model, the reference image is encoded into a latent feature vector, and the training prompt information is converted into an embedded feature vector; a random noise vector is generated that matches the vector dimension of the latent feature vector; the random noise vector is added to the latent feature vector according to a preset time step; and an output image is obtained based on the latent feature vector with the added random noise vector and the embedded feature vector.

[0166] Based on the global features of the output image and the reference image, a first loss value is obtained; wherein the first loss value is used to indicate whether the output image contains the global features of the reference image; based on the specified information features of the output image and the reference image, a second loss value is obtained; wherein the second loss value is used to indicate whether the output image contains the specified information features of the reference image; and a final loss value is determined based on the first loss value and the second loss value.

[0167] Based on the preset weight parameters, the weighted sum of the first loss value and the second loss value is determined as the final loss value.

[0168] The specified image information includes: specified image content and specified image style; based on the specified image content of the output image and the reference image, a first sub-loss value is obtained; wherein the first sub-loss value is used to: indicate whether the output image contains the specified image content of the reference image; based on the specified image style of the output image and the reference image, a second sub-loss value is obtained; wherein the second sub-loss value is used to: indicate whether the output image contains the specified image style of the reference image; based on the first sub-loss value and the second sub-loss value, a second loss value is determined.

[0169] In the above method, based on the parameter matrix corresponding to the specified image information, the specified model parameters in the image generation model are adjusted, and the target image is output through the image generation model after parameter adjustment. This method improves the image migration effect, avoids overfitting, and can achieve precise control of detail information and overall information.

[0170] This embodiment further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned image generation method.

[0171] The computer-executable instructions stored in the computer-readable storage medium can implement the following operations in the above-mentioned image generation method by executing the computer-executable instructions:

[0172] Determine a parameter matrix corresponding to specified image information; wherein the specified image information includes: specified image content and / or specified image style; load the parameter matrix into a pre-trained image generation model, and adjust specified model parameters in the image generation model through the parameter matrix; input preset basic information into the image generation model after parameter adjustment, and output a target image; wherein a first portion of image information of the target image matches the specified image information, and a second portion of image information of the target image matches the basic information; the basic information includes: text prompts and / or a basic image.

[0173] The parameter matrix is ​​preset with a matrix index, which is used to: indicate that the parameter matrix is ​​loaded into a specified model position in the image generation model; load the parameter matrix into the specified model position indicated by the matrix index in the pre-trained image generation model, and adjust the specified model parameters in the specified model position through the parameter matrix.

[0174] The matrix index is specifically used to: indicate that the parameter matrix is ​​loaded into a specified feature conversion module in the image generation model; load the parameter matrix into the specified feature conversion module indicated by the matrix index in the pre-trained image generation model, and adjust the module parameters in the specified feature conversion module through the parameter matrix.

[0175] The specified feature conversion module includes: a self-attention layer and a cross-attention layer; the self-attention layer and / or the cross-attention layer in the specified feature conversion module is adjusted through a parameter matrix.

[0176] The specified image information includes multiple types; different specified image information corresponds to different specified feature conversion modules with different parameter matrix settings.

[0177] The specified image information includes the specified image content and the specified image style; the intermediate-level feature conversion module in the U-Net network of the image generation model; the specified feature conversion module with the parameter matrix setting corresponding to the specified image style, including: the feature conversion module in the downsampling path or the upsampling path in the U-Net network of the image generation model.

[0178] The parameter matrix is ​​obtained through pre-training, and the parameter matrix includes: an information capture submatrix and an information expression submatrix; determining the matrix product of the information capture submatrix and the information expression submatrix; wherein the dimension of the matrix product matches the dimension of the original parameters of the specified model parameters; superimposing the matrix product on the original parameters of the specified model parameters to obtain the specified model parameters in the adjusted image generation model.

[0179] The parameter matrix corresponding to the specified image information is trained in the following manner: loading an initial matrix into a specified feature conversion module in an image generation model; wherein the initial matrix is ​​used to adjust module parameters of the specified feature conversion module; inputting a preset reference image and preset training prompt information into the image generation model to obtain an output image; wherein the reference image contains the specified image information; determining a loss value based on the output image and the reference image; adjusting matrix parameters of the initial matrix based on the loss value until the first part of the image information of the output image matches the specified image information, and determining the adjusted initial matrix as the parameter matrix.

[0180] The reference image is a single image.

[0181] The initial matrix includes: a first initial submatrix and a second initial submatrix; wherein the first initial submatrix corresponds to the information capture submatrix in the parameter matrix; the second initial submatrix corresponds to the information expression submatrix in the parameter matrix; the initial values ​​of the parameters of the first initial submatrix are Gaussian distribution, and the initial values ​​of the parameters of the second initial submatrix are zero.

[0182] A preset reference image and preset training prompt information are input into an image generation model. Through the image generation model, the reference image is encoded into a latent feature vector, and the training prompt information is converted into an embedded feature vector; a random noise vector is generated that matches the vector dimension of the latent feature vector; the random noise vector is added to the latent feature vector according to a preset time step; and an output image is obtained based on the latent feature vector with the added random noise vector and the embedded feature vector.

[0183] Based on the global features of the output image and the reference image, a first loss value is obtained; wherein the first loss value is used to indicate whether the output image contains the global features of the reference image; based on the specified information features of the output image and the reference image, a second loss value is obtained; wherein the second loss value is used to indicate whether the output image contains the specified information features of the reference image; and a final loss value is determined based on the first loss value and the second loss value.

[0184] Based on the preset weight parameters, the weighted sum of the first loss value and the second loss value is determined as the final loss value.

[0185] The specified image information includes: specified image content and specified image style; based on the specified image content of the output image and the reference image, a first sub-loss value is obtained; wherein the first sub-loss value is used to: indicate whether the output image contains the specified image content of the reference image; based on the specified image style of the output image and the reference image, a second sub-loss value is obtained; wherein the second sub-loss value is used to: indicate whether the output image contains the specified image style of the reference image; based on the first sub-loss value and the second sub-loss value, a second loss value is determined.

[0186] In the above method, based on the parameter matrix corresponding to the specified image information, the specified model parameters in the image generation model are adjusted, and the target image is output through the image generation model after parameter adjustment. This method improves the image migration effect, avoids overfitting, and can achieve precise control of detail information and overall information.

[0187] The computer program products of the image generation method, device, and electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.

[0188] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0189] In addition, in the description of the embodiments of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0190] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0191] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0192] Finally, it should be noted that the above embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An image generation method, characterized in that: The method comprises: Determining a parameter matrix corresponding to specified image information; wherein the specified image information includes: specified image content and / or specified image style; Loading the parameter matrix into a pre-trained image generation model, and adjusting specified model parameters in the image generation model using the parameter matrix; The preset basic information is input into the image generation model after parameter adjustment, and a target image is output; wherein the first part of the image information of the target image matches the specified image information, and the second part of the image information of the target image matches the basic information; the basic information includes: text prompts and / or basic images.

2. The method according to claim 1, characterized in that The parameter matrix is ​​preset with a matrix index, and the matrix index is used to: indicate that the parameter matrix is ​​loaded into a specified model position in the image generation model; The step of loading the parameter matrix into a pre-trained image generation model and adjusting the specified model parameters in the image generation model using the parameter matrix includes: The parameter matrix is ​​loaded into the specified model position indicated by the matrix index in the pre-trained image generation model, and the specified model parameters in the specified model position are adjusted by the parameter matrix.

3. The method according to claim 2, characterized in that The matrix index is specifically used to: instruct the parameter matrix to be loaded into a specified feature conversion module in the image generation model; The step of loading the parameter matrix into the designated model position indicated by the matrix index in the pre-trained image generation model, and adjusting the designated model parameters in the designated model position by using the parameter matrix comprises: The parameter matrix is ​​loaded into the specified feature conversion module indicated by the matrix index in the pre-trained image generation model, and the module parameters in the specified feature conversion module are adjusted by the parameter matrix.

4. The method according to claim 3, characterized in that The specified feature conversion module includes: a self-attention layer and a cross-attention layer; The step of adjusting the module parameters in the specified feature conversion module through the parameter matrix includes: adjusting the self-attention layer and / or cross-attention layer in the specified feature conversion module through the parameter matrix.

5. The method according to claim 3, characterized in that The designated image information includes multiple types; different designated image information corresponds to different designated feature conversion modules set by parameter matrices.

6. The method according to claim 3, characterized in that The specified image information includes specified image content and specified image style; The specified feature conversion module of the parameter matrix setting corresponding to the specified image content includes: an intermediate layer feature conversion module in the U-Net network of the image generation model; The specified feature conversion module set by the parameter matrix corresponding to the specified image style includes: a feature conversion module in the downsampling path or the upsampling path in the U-Net network of the image generation model.

7. The method according to claim 1, characterized in that The parameter matrix is ​​obtained through pre-training, and the parameter matrix includes: an information capture sub-matrix and an information expression sub-matrix; The step of adjusting the specified model parameters in the image generation model by using the parameter matrix comprises: Determining a matrix product of the information capture submatrix and the information expression submatrix; wherein the dimension of the matrix product matches the dimension of the original parameters of the specified model parameters; The matrix product is superimposed on the original parameters of the specified model parameters to obtain the adjusted specified model parameters in the image generation model.

8. The method according to claim 1, characterized in that The parameter matrix corresponding to the specified image information is trained in the following way: Loading an initial matrix into a specified feature conversion module in the image generation model; wherein the initial matrix is ​​used to adjust module parameters of the specified feature conversion module; Inputting a preset reference image and preset training prompt information into the image generation model to obtain an output image; wherein the reference image includes the specified image information; determining a loss value based on the output image and the reference image; Matrix parameters of the initial matrix are adjusted based on the loss value until the first portion of image information of the output image matches the specified image information, and the adjusted initial matrix is ​​determined as a parameter matrix.

9. The method according to claim 8, characterized in that The reference image is a single image.

10. The method according to claim 8, characterized in that The initial matrix includes: a first initial sub-matrix and a second initial sub-matrix; The first initial sub-matrix corresponds to the information capture sub-matrix in the parameter matrix; the second initial sub-matrix corresponds to the information expression sub-matrix in the parameter matrix; Initial values ​​of parameters of the first initial submatrix are Gaussian distribution, and initial values ​​of parameters of the second initial submatrix are zero.

11. The method according to claim 8, characterized in that The step of inputting a preset reference image and preset training prompt information into the image generation model to obtain an output image includes: Inputting a preset reference image and preset training prompt information into the image generation model, encoding the reference image into a latent feature vector and converting the training prompt information into an embedded feature vector through the image generation model; generating a random noise vector that matches the vector dimension of the latent feature vector; Adding the random noise vector to the latent feature vector according to a preset time step; An output image is obtained based on the latent feature vector to which the random noise vector is added and the embedded feature vector.

12. The method according to claim 8, characterized in that The step of determining a loss value based on the output image and the reference image comprises: Obtaining a first loss value based on the global features of the output image and the reference image; wherein the first loss value is used to indicate whether the output image contains the global features of the reference image; Obtaining a second loss value based on the specified information features of the output image and the reference image; wherein the second loss value is used to indicate whether the output image contains the specified information features of the reference image; A final loss value is determined based on the first loss value and the second loss value.

13. The method according to claim 12, characterized in that The step of determining a final loss value based on the first loss value and the second loss value comprises: Based on a preset weight parameter, a weighted sum of the first loss value and the second loss value is determined as a final loss value.

14. The method according to claim 12, characterized in that The specified image information includes: specified image content and specified image style; The step of obtaining a second loss value based on the specified information features of the output image and the reference image comprises: Obtaining a first sub-loss value based on the output image and the specified image content of the reference image; wherein the first sub-loss value is used to: indicate whether the output image contains the specified image content of the reference image; Obtaining a second sub-loss value based on the output image and the specified image style of the reference image; wherein the second sub-loss value is used to: indicate whether the output image contains the specified image style of the reference image; A second loss value is determined based on the first sub-loss value and the second sub-loss value.

15. An image generating device, characterized in that: The device comprises: A parameter matrix determination module is used to determine a parameter matrix corresponding to the specified image information; wherein the specified image information includes: specified image content and / or specified image style; A model parameter adjustment module, configured to load the parameter matrix into a pre-trained image generation model and adjust specified model parameters in the image generation model using the parameter matrix; A target image output module is used to input preset basic information into the image generation model after parameter adjustment and output a target image; wherein the first part of the image information of the target image matches the specified image information, and the second part of the image information of the target image matches the basic information; the basic information includes: text prompts and / or basic images.

16. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the image generation method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the image generation method according to any one of claims 1 to 14.

Images