Multi-style image generation method and image generation device based on large model

Abstract

The present application relates to the field of deep learning technology, and proposes a multi-style image generation method and image generation device based on a large model, including: constructing a training set for each style; using a contrastive learning model to obtain a deep feature vector based on positive and negative samples; using a semantic segmentation network to obtain a true label and a semantic label vector; determining a connected structure graph based on feature points, and determining a structural feature vector based on a semantic label vector and a connected structure graph; determining a content-structure feature vector based on a deep feature vector and a structural feature vector; classifying the training set based on the similarity of images in a bucket during model training; determining the sample sampling order and the alternating order between buckets in model training based on the classification results, and then completing the SDXL model training; and generating a style image using a model deployed in the cloud according to the user's imaging needs. The present application improves the matching degree between style images and needs by reducing the probability of confusion caused by the model.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a multi-style image generation method and image generation device based on a large model. Background Art

[0002] With the increasing demand for personalization and customization by users, AI imaging technology is becoming more and more popular on AI imaging devices. AI imaging devices are composed of touch screens, cameras, fill lights, thermal sublimation printers, motherboards and other components, and are imaging devices that can output a large number of images in different styles. At this stage, AI imaging devices have been applied to the generation of images in various styles, such as children's AI portraits, artistic portraits, professional photos, and ID photos. Users select different styles on AI imaging devices to provide corresponding prompt words, and the generation model generates images of corresponding styles based on the prompt words and user images.

[0003] At present, the popular multi-style image generation models can be divided into two categories: text-generated images and image-generated images. The mainstream neural network models include StyleGAN, Stable Diffusion XL, etc. When learning multi-style images, the generation model is prone to confusion when facing samples containing multiple objects or subjects, resulting in the generated style image being consistent with expectations as a whole, but the attributes of different objects in the image are cross-replaced, such as color swapping, resulting in a deviation between the generated image results and user needs. Summary of the invention

[0004] This application provides a multi-style image generation method and image generation device based on a large model to solve the problem that the generated style image deviates from the user's needs due to the poor effect of the model in processing multi-object style images. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating multi-style images based on a large model, the method comprising the following steps:

[0006] Use the de-noised images in each style to build a training set for each style;

[0007] The images in the training set are divided into positive and negative samples, and a deep feature vector of the content of each image is obtained using a contrastive learning model;

[0008] Taking the training set of each style as input, a semantic segmentation network is used to determine the true label of each pixel in each image, the semantic label vector of the feature point, and the structural feature atlas of the attribute;

[0009] Determine a structural feature vector of a feature point based on the semantic label vector of the feature point and the structural feature atlas using a graph attention network;

[0010] Determine a content-structure feature vector for each image based on the deep feature vector and the structural feature vector;

[0011] Classifying the training set using a graph clustering algorithm based on the similarity of images in the bucket during the training of the U-Net model in the SDXL model;

[0012] Based on the classification results, the sample sampling order and the alternating order between buckets in the U-Net model training process in the SDXL model are respectively determined, and the content-structure feature vector is used as an additional embedding condition to complete the training of the U-Net model based on the sampling order and the alternating order;

[0013] The AI ​​imaging device communicates with the server according to the user's imaging needs, uses the trained SDXL model to generate a style image, and sends it to the AI ​​imaging device.

[0014] In the above scheme, positive and negative samples are constructed through the image segmentation results of each style, and the contrastive learning model is used to obtain the deep feature vector of each image about the content. Using different image blocks of an image as positive samples in the contrastive learning model can obtain network features with higher granularity for each image, so that images of the same style can also be expressed differently according to the detailed local features of the image content; then, the layout structure features of the image under each style contained in the structural feature graph composed of feature points in the connected domain of different attributes in the image are learned through the graph attention network to obtain the deep feature vector of the structure of each image, and the content-structure feature vector of each image is determined after splicing and fusion, so as to realize the joint representation of the content and layout structure of images of different styles; then, the clustering of the training set samples is realized based on the content-structure feature vector and the similarity between the images as a whole, and the classification is based on The clustering results determine the sampling rules of samples in each bucket and the exchange order between buckets during U-Net training. The samples for each round of U-Net training are determined by the sample sampling rules and the exchange order between buckets, which reduces the network fluctuation caused by the large difference in attribute categories between adjacent sampled samples during model training, so that the model can be trained in an orderly manner. Finally, the content-structure feature vector of the image in the training set is added to the original Embedding through Fourier encoding to obtain additional style image structure conditions, so that the potential features learned by U-Net can better learn the content information and structural features of images in different style training sets, thereby reducing the probability of confusion when the generated model SDXL faces samples containing multiple objects or subjects, and improving the matching degree between the style image output by the AI ​​imaging device and user needs.

[0015] In combination with the first aspect, in some possible implementations, segmenting the images in the training set into positive and negative samples, and obtaining a deep feature vector of the content of each image using a contrastive learning model includes:

[0016] For each style of training set, use the image segmentation algorithm to divide each image into multiple image blocks, and use all the image blocks of one image as positive samples, and use all the image blocks of the remaining images as negative samples;

[0017] The positive and negative samples are taken as input, and the feature vector output by the contrastive learning Moco model is used as the deep feature vector of the content of the image corresponding to the positive sample.

[0018] In the above scheme, each image is divided into a large number of image blocks by image segmentation, and the image blocks of each image and the image blocks of other images are used as positive and negative samples respectively. The difference between the positive and negative samples is learned by the contrast learning model, so that the image features of each image can be represented more finely and distinctively, that is, the deep feature vector of the content of each image is obtained, so that the content-structure feature vector of each image can be obtained by vector splicing later, and then the additional embedding conditions when training U-Net are determined.

[0019] In combination with the first aspect above, in some possible implementations, the using of a semantic segmentation network to respectively determine the true labels of pixels in each image, the semantic label vectors of feature points, and the structural feature atlas of attributes includes:

[0020] Taking the training set of each style as input, using the semantic segmentation network to obtain all semantic labels of each pixel in each image in the training set and the confidence of each semantic label, and taking the semantic label with the highest confidence as the true label of each pixel;

[0021] Pixels with the same true label in each image in the training set are regarded as a class of pixels with the same attributes, a connected domain of each class of pixels is determined, all feature points in the connected domain are determined using a feature point detection algorithm, and a vector composed of semantic labels at all confidence levels of each feature point is used as a semantic label vector of each feature point;

[0022] The structural feature atlas of each attribute is determined based on the connected domain extraction results of the feature points.

[0023] In the above scheme, a semantic segmentation network is used to determine the semantic labels of all confidence levels for each pixel in the image, and the true label of the pixel is determined based on the semantic label; secondly, feature point detection is used to extract feature points in each weekly style image, and the semantic label vector of the feature points is further determined to facilitate the subsequent extraction of structural features of the image under each style.

[0024] In combination with the first aspect above, in some possible implementations, determining a structural feature atlas of each attribute based on a connected domain extraction result of feature points includes:

[0025] The undirected graph formed by connecting all feature points in each connected domain in each image is taken as the connected structure graph of each connected domain, and the set of connected structure graphs of all connected domains under each attribute is taken as the structural feature atlas of each attribute.

[0026] In the above scheme, the connectivity structure graph of each connected domain is constructed through the feature points in each connected domain, and then the structural feature atlas of each attribute in each image is obtained, so that the subsequent graph attention network can learn the structural features of images in different styles.

[0027] In combination with the first aspect, in some possible implementations, the determining the structural feature vector of the feature point based on the semantic label vector of the feature point and the structural feature atlas using the graph attention network includes:

[0028] The semantic label vector of each feature point in each connected domain is used as the initial feature vector of each feature point in the connected structure graph of each connected domain;

[0029] The connected structure graph after setting the initial feature vector in the structural feature atlas of all attributes in each image is taken as input, and the structural feature vector of each feature point is obtained using the graph attention network.

[0030] In the above scheme, the semantic label vector of the feature point and the connected structure graph are used as input, and the graph attention network is used to better learn the layout structural features of each style, so as to obtain a quantitative expression of the structural features of each style image, that is, to obtain the structural feature vector of each feature point, so as to obtain the content-structure feature vector of each image through vector splicing, so that the subsequent U-Net model can better learn the layout structural features at the image feature point level under each style, and improve the matching degree between objects and attributes in the generated style image.

[0031] In combination with the first aspect above, in some possible implementations, the method for determining the content-structure feature vector of each image based on the deep feature vector and the structure feature vector is:

[0032] A set consisting of structural feature vectors of all feature points in each connected domain in each image is determined, and multiple sets consisting of structural feature vectors of all feature points in all connected domains in each image are determined;

[0033] In the order of the deep feature vector about the content of each image first and the multiple sets second, the deep feature vector about the content of each image and the multiple sets are concatenated as the content-structure feature vector of each image.

[0034] In the above scheme, the structural feature vectors of each feature point are used to form a set to express the structural features of the connected domain, and then the structural feature expressions of all connected domains are used together with the expression of the image content. That is, the multiple sets are sequentially spliced ​​with the deep feature vectors to obtain the content-structure feature vector of each image, thereby achieving effective expression of the image content of each image in the training set of each style and the layout structural features of each style.

[0035] In combination with the first aspect, in some possible implementations, the classifying the training set using a graph clustering algorithm based on the similarity of images in the bucket during the training of the U-Net model in the SDXL model includes:

[0036] The training set of each style is bucketed based on the aspect ratio;

[0037] The similarity measurement result between each image in each bucket and any image in another bucket is taken as the first similarity;

[0038] The similarity measurement result of the content-structure feature vector between each image in each bucket and any image in another bucket is used as the second similarity, and the ratio of the second similarity to the first similarity is used as the main similarity between the two images;

[0039] Each image is taken as a node, the subject similarity between images is taken as the weight between nodes, the undirected graph constructed using all images in the training set of each style is taken as input, and the graph clustering algorithm is used to obtain the clustering cluster where the image in each bucket is located.

[0040] In the above scheme, clustering of training set samples is achieved based on content-structure feature vectors and the similarity between the images as a whole. The similarity of the main contents of the two images is evaluated through the first similarity and the second similarity, so as to accurately evaluate the similarity of key contents between images. Secondly, the classification results of images in each bucket with complex contents are made better through graph clustering, so as to determine the order of sampling samples in each bucket during U-Net training.

[0041] In combination with the first aspect above, in some possible implementations, determining the sample sampling order and the alternating order between buckets in the U-Net model training process in the SDXL model based on the classification results respectively includes:

[0042] During each training process of the U-Net model, a preset number of images belonging to the same cluster are sampled in each bucket as input for training;

[0043] When switching between buckets, the set of images corresponding to the cluster centers of the clusters to which all samples in each bucket belong is used as the category representative set of each bucket;

[0044] Calculate the similarity between the category representative sets of two buckets, arrange the similarity between each bucket and the category representative sets of other buckets in descending order to obtain a sequence as the alternating sequence sequence of each bucket, and when sampling is alternately exchanged between buckets, follow the order in the alternating sequence sequence.

[0045] In the above scheme, based on the clustering results, the sampling rules of samples in each bucket and the exchange order between buckets during U-Net training are determined in turn. The samples for each round of U-Net training are determined by the sample sampling rules and the exchange order between buckets, which reduces the network fluctuation caused by the large difference in attribute categories between adjacent sampled samples during model training, so that the model can be trained in an orderly manner.

[0046] In combination with the first aspect, in some possible implementations, the training of the U-Net model based on the sampling order and the alternating order using the content-structure feature vector as an additional embedding condition includes:

[0047] Perform Fourier encoding on the content-structure feature vector of each image to obtain the corresponding Embedding feature, embed the Embedding feature into Time-Embedding, and input it into U-Net as an additional style image structure condition;

[0048] Time-Embedding is used to train the U-Net model based on the sampling rules within each bucket and the alternating order between buckets.

[0049] In the above scheme, the content-structure feature vector of each image is embedded into the Embedding feature by encoding as an additional embedding condition during U-Net model training. The feature vectors representing the content features and structural features of each style image are used to enrich the original embedding information of the generative model SDXL model, thereby improving the fine-grained learning of U-Net model for images of different styles, thereby reducing the probability of confusion when SDXL faces samples containing multiple objects or subjects.

[0050] In combination with the first aspect above, in some possible implementations, the AI ​​imaging device communicates with the cloud according to the user's imaging requirements, generates a style image using the trained SDXL model, and sends the style image to the AI ​​imaging device, including:

[0051] Deploy the trained SDXL model on the cloud;

[0052] The user inputs personal requirements on the AI ​​imaging device, and the AI ​​imaging device receives the personal requirements and sends them to the cloud;

[0053] The SDXL model is used to generate style images that meet the user's personal needs, and the cloud transmits the generated style images back to the AI ​​imaging device.

[0054] In the above scheme, the trained SDXL model is deployed to the cloud so that different AI imaging devices can send the user's personal needs to the cloud, and use the SDXL model to generate style images matching the needs, so that the AI ​​imaging device outputs style images based on the user's needs.

[0055] In a second aspect, a device for generating a multi-style image based on a large model is provided, the device comprising:

[0056] A preprocessing module is used to construct a training set for each style using the de-noised images in each style;

[0057] A first extraction module is used to divide the images in the training set into positive and negative samples, and obtain a deep feature vector of the content of each image using a contrastive learning model;

[0058] The semantic extraction module is used to take the training set of each style as input and use the semantic segmentation network to determine the true label of the pixel points in each image, the semantic label vector of the feature points, and the structural feature atlas of the attributes;

[0059] A second extraction module is used to determine a structural feature vector of a feature point based on the semantic label vector of the feature point and the structural feature atlas using a graph attention network;

[0060] A splicing module, configured to determine a content-structure feature vector of each image based on the deep feature vector and the structure feature vector;

[0061] A classification module, for classifying the training set using a graph clustering algorithm based on the similarity of images in the bucket during the training of the U-Net model in the SDXL model;

[0062] A training module, for determining the sample sampling order and the alternating order between buckets in the U-Net model training process in the SDXL model based on the classification results, and completing the training of the U-Net model based on the sampling order and the alternating order by using the content-structure feature vector as an additional embedding condition;

[0063] The imaging module is used for the AI ​​imaging device to communicate with the server according to the user's imaging requirements, and uses the trained SDXL model to generate style images and send them to the AI ​​imaging device.

[0064] In a third aspect, a server is provided, comprising a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the device executes the method in the first aspect or any possible implementation of the first aspect.

[0065] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0067] Figure 1 A schematic diagram of an implementation environment of a large model-based multi-style image generation method provided by an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a flow chart of a method for generating multi-style images based on a large model provided in one embodiment of the present application;

[0069] Figure 3 A flowchart for implementing the contrastive learning model provided in one embodiment of the present application to extract deep feature vectors of image content;

[0070] Figure 4 A schematic diagram of a connected structure diagram provided for an embodiment of the present application;

[0071] Figure 5 A schematic diagram of the structure of a large model-based multi-style image generation device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0074] The implementation environment of the embodiment of the present application is introduced as follows. Figure 1 As shown, the implementation environment of the large model-based multi-style image generation method provided in the embodiment of the present application includes a data processing end 101, a server 102, a training end 103, and an AI imaging device 104.

[0075] The data processing end 101 and the training end 103 are connected via a wireless network, and the AI ​​imaging device 104 and the server 102 are connected via a wireless or wired network. The data processing end 101 obtains the image of each style and performs denoising to construct a training set of each style; and respectively uses the contrast learning model and the graph attention network to extract the image's deep feature vector about the content and the structural feature vector of the feature points in the image, and the two types of feature vectors are spliced ​​to obtain the content-structure feature vector of each image; the data processing end 101 uploads the training set and the content-structure feature vector to the training end 103, and the training end 103 buckets the images in the training set based on the bucketing strategy, and then uses the similarity of the content-structure feature vectors between the images and the similarity between the images to calculate the distance between the two images. The subject similarity is determined, and the images are divided into different clusters based on the subject similarity. The sampling rules in each bucket and the replacement order between buckets during U-Net training are determined based on the clustering results; and the content-structure feature vector is used as an additional embedding during training to complete the training. The training end 103 deploys the trained generation model SDXL to the server 102, and the AI ​​imaging device 104 uploads the style information and personal images selected by the user on the imaging device to the server 102. The server 102 outputs the style image using the deployed generation model and transmits the generated style image back to the AI ​​imaging device 104.

[0076] Server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks, as well as big data and artificial intelligence platforms.

[0077] The following is an introduction to the technical solution provided by the embodiment of the present application. The embodiment of the present application provides a multi-style image generation method based on a large model. Please refer to Figure 2 , Figure 2 : is a schematic diagram of the implementation process of the multi-style image generation method based on a large model provided in an embodiment of the present application, the method comprising the following steps:

[0078] 201, the de-noised images in each style are used to construct training sets for each style.

[0079] In this application, in order to enable the AI ​​generation device to output style images for different needs of users, a large number of existing images in each style are first collected as the initial data source, and the number of images collected in each style is the same. Among them, the styles include but are not limited to ID photos, such as ID photos with a blue background; artistic photos, such as landscape photos and costume photos; job-seeking photos, such as half-length formal photos with a white background; professional image photos, such as doctors, lawyers and teachers; superheroes, such as Iron Man, Spiderman and Doctor Strange.

[0080] For each captured image in each style, an image denoising algorithm is used for processing in order to improve the image quality, thereby improving the quality of subsequent model samples. In this application, image denoising algorithms include two types: traditional image denoising methods and deep learning denoising methods. Traditional image denoising methods include but are not limited to median filtering denoising, bilateral filtering denoising, and non-local mean NL-Means algorithm; deep learning denoising models include but are not limited to DnCNN (Denoising Convolutional Neural Network) and AD-Net (Attention-guided CNN). Image denoising is a well-known technology and the specific process will not be repeated. All denoised images in each style are used to form a training set for each style.

[0081] 202 , dividing the images in the training set into positive and negative samples, and using a contrastive learning model to obtain a deep feature vector of the content of each image.

[0082] Here, the deep feature vector about the content is the expression of the image content in each image output by the contrastive learning model. In this application, firstly, each image in each training set is segmented into image blocks by image segmentation. The image segmentation method includes but is not limited to superpixel segmentation algorithm, threshold segmentation algorithm, and watershed segmentation algorithm. Secondly, based on the segmented image blocks, the contrastive learning method is used to learn the content in the more detailed image blocks in each image, so as to avoid the generation model ignoring the detailed features when facing large-scale images during training. The process of obtaining the deep feature vector about the content is as follows: Figure 3 As shown:

[0083] In one embodiment, each image is taken as input, and each image is divided into multiple image blocks using a simple linear iterative segmentation SLIC algorithm. The number of seed points in the SLIC algorithm is set to K. In this embodiment, the value of K is 100. The SLIC algorithm is a well-known technology, and the specific process is not repeated here.

[0084] In another embodiment, each image is taken as input and each image is divided into a plurality of image blocks using a watershed segmentation algorithm. The watershed algorithm is a well-known technology and the specific process will not be described in detail.

[0085] It should be noted that if the image blocks after image segmentation are irregular in shape and have different scales, they are scaled uniformly based on the maximum length and maximum width in the image blocks to facilitate subsequent use as model input. Image scaling is a well-known technology and the specific process will not be repeated here.

[0086] Furthermore, the image blocks segmented from each image in the training set of each style are taken as positive samples, and the image blocks segmented from the remaining images in the training set of each style are taken as negative samples. The purpose of dividing the positive and negative samples in this way is to obtain more fine-grained network features for each image by taking different image blocks of images in the same style as positive and negative samples, so that images of the same style can also be expressed differently according to the different detailed local features of the image content.

[0087] Secondly, for each image, the corresponding positive samples and negative samples are used as input, and the network feature vector output by the feature extraction layer in the contrastive learning Moco model is used as the deep feature vector of each image about the content. The optimization algorithm in the model is the Adam (Adaptive moment estimation) algorithm, and the loss function is InfoNCE (Information Noise Contrastive Estimation loss). The training of the neural network is a well-known technology, and the specific process will not be repeated here.

[0088] 203, taking the training set of each style as input, a semantic segmentation network is used to determine the true label of the pixel in each image, the semantic label vector of the feature point, and the structural feature atlas of the attribute.

[0089] For images of different styles, there are obvious differences in the spatial layout, structure, color and other information in the images. For example, the theme colors in costume photos and portrait photos may be similar, but the clothing, environment, etc. in the images will have obvious style differences. If the model training focuses only on one aspect of style and spatial layout, it will cause the model to be easily confused when facing samples containing multiple objects or subjects. Therefore, in this application, by processing the pixels of the same attribute in each image, the structural features of the area corresponding to each attribute are obtained, and the deep feature vector of the image about the content and the structural features of different attributes in the image are used to form a spliced ​​feature vector related to each style, so as to realize the joint representation of the image content features and structural features.

[0090] Specifically, the training set of each style is used as input, and the semantic segmentation network is used to output the semantic labels of each pixel under all confidence levels. The loss function of the semantic segmentation network is the cross entropy loss function, and the optimizer is the Adam optimizer. The semantic label corresponding to the maximum confidence level of each pixel in each image is used as the true label of each pixel, and all pixels with the same true label in each image are used as pixels with the same attributes. Under the premise that the semantic segmentation of the images in the training set can be achieved, the semantic segmentation network includes but is not limited to Deeplab, CGnet, and SegNext. This application does not impose any special restrictions on the type of semantic segmentation network.

[0091] Secondly, take each type of pixel points with the same attribute as input, and use the connected domain extraction algorithm to obtain all connected domains with the same attribute; then use feature point detection to determine the feature points in each connected domain. In this application, the purpose of feature point detection is to extract feature points in connected domains. Therefore, under the premise that the purpose of feature point detection can be achieved, the feature point detection method includes but is not limited to SIFT feature point detection, FAST feature point detection, and ORB feature point detection. The vector composed of all the confidence semantic labels of each feature point in each image is used as the semantic label vector of each feature point. This application does not impose any special restrictions on the feature point detection method.

[0092] Furthermore, all feature points in each connected domain are connected to obtain a connected structure graph of each connected domain. It should be noted that the feature points are connected in a manner that two adjacent feature points are connected in sequence to form a closed connected structure graph, such as Figure 4As shown in the figure, the left figure is the detection result of feature points in a connected domain, and the right figure is the connected structure diagram corresponding to this connected domain. The set of connected structure diagrams of all connected domains under each attribute in each image is taken as the structural feature atlas of each attribute.

[0093] 204 , using a graph attention network to determine a structural feature vector of the feature point based on the semantic label vector of the feature point and the structural feature atlas.

[0094] Here, the structural feature vector is the expression of the image layout structure of each feature point in each image output by the graph attention network. In this application, the input of the graph attention network is first determined using the semantic label vector of the feature point on each connected domain and the connected structure graph of the connected domain. Secondly, the graph attention network is used to learn the layout structural features in each image under each style, so as to facilitate the joint representation with the deep feature vector of the image about the content, and finally as an additional embedding condition during the U-Net training in the generative model, so that the potential features output by the U-Net can have better structural features in the style image. The process of obtaining the structural feature vector is as follows:

[0095] First, the semantic label vector of each feature point in each connected domain is used as the initial feature vector of each feature point in the connected structure graph of each connected domain, and the semantic label vectors of all feature points on the connected domain are used to process the connected structure graph of the connected domain. The purpose of this is to enable the graph attention network to focus on semantics and structure. Secondly, the connected structure graph after setting the initial feature vector in the structural feature atlas of all attributes in each image is used as input, and the feature vector weighted by the mutual correlation coefficient of attention is obtained by using the graph attention network GAT as the structural feature vector of each corresponding feature point. It should be noted that in this application, the calculation of the attention mechanism in GAT is only performed on neighbor vertices. This is because in actual situations, in images of most styles, there are almost no structural features in which feature points are randomly connected in a connected domain. The loss function of GAT is negative log-likelihood loss NLLloss, the optimizer is Adam optimizer, and the training of the neural network is a well-known technology, and the specific process will not be repeated.

[0096] 205 : Determine a content-structure feature vector of each image based on the deep feature vector and the structure feature vector.

[0097] Here, the content-structure feature vector is obtained by concatenating the deep feature vectors about the content of each image obtained in the above steps and the structural feature vectors of the feature points in each image. It can jointly represent the content and layout structure of each image in each style, and thus add this as an additional condition related to the style to the U-Net training in the generation model.

[0098] Specifically, a set of structural feature vectors of all feature points in each connected domain in each image is formed to realize the collective representation of the entire connected domain; secondly, in the order of the deep feature vector of the content of each image in front and the set of structural feature vectors of feature points in all connected domains in each image in the back, the deep feature vector of the content of each image and the splicing result of the multiple sets are used as the content-structure feature vector of each image. Among them, the content-structure feature vector of the i-th image is T i =(t i ,t i,1 ,t i,2 ,…,t i,n ), where t i is the deep feature vector of the content of the ith image, n is the number of connected structure graphs in the ith image, t i,1 ,t i,2 With t i,n are the sets of structural feature vectors of all feature points on the first, second and nth connected structure graphs in the i-th image, respectively, and t i,1 =(t i,1 (1),t i,1 (2),…,t i,1 (m)), where t i,1 (1) t i,1 (2) with t i,1 (m) are the structural feature vectors of the first, second and mth feature points on the first connected structure graph in the i-th image, respectively, and m is the number of feature points on the first connected structure graph in the i-th image.

[0099] 206 , classifying the training set using a graph clustering algorithm based on the similarity of the images in the bucket during the training of the U-Net model in the SDXL model.

[0100] Here, the purpose of classifying the training set is that when the generation model SDXL is trained, it is usually divided into buckets based on the aspect ratio of the sample, and then a sample in a bucket is randomly selected for training. However, due to the uncertainty between the content and image structure of the randomly selected sample, this application selects samples from the bucket in an orderly manner based on the classification results of the images in the training set, so as to improve the model's ability to learn the subordinate relationship between attributes and objects in each style image.

[0101] Specifically, when training U-Net based on the training set of each style, bucketing is performed based on the aspect ratio of the image. First, the similarity measurement result A between each image in each bucket and any image in another bucket is calculated. In this embodiment, the structural similarity SSIM between the two images is calculated as the similarity measurement result A; secondly, the similarity measurement result B between each image in each bucket and the content-structure feature vector of any image in another bucket is calculated. In this embodiment, the Pearson correlation coefficient between the two vectors is calculated as the similarity measurement result B; the ratio of the similarity measurement result B to the similarity measurement result A is used as the main similarity between each image in each bucket and any image in another bucket. Among them, the calculation of the structural similarity SSIM and the Pearson correlation coefficient is a well-known technology, and the specific process is not repeated.

[0102] It should be noted that this embodiment only provides a similarity measurement method between images and a similarity measurement method between content-structure feature vectors, which are respectively calculating the structural similarity SSIM between images and calculating the Pearson correlation coefficient between content-structure feature vectors. On the premise that the similarity measurement between images and the similarity measurement between vectors can be respectively realized, in other embodiments, other measurement methods can also be selected to obtain the similarity measurement results between images and the similarity measurement results between content-structure feature vectors, respectively.

[0103] Further, each image is taken as a node, the subject similarity between the two images is taken as the connection weight between the corresponding two nodes, and the weighted undirected graph composed of all images is taken as input, and the weighted undirected graph is divided into multiple clusters using a graph clustering algorithm. The graph clustering algorithm is a well-known technology, and the specific process is not repeated. It should be understood that under the premise of being able to achieve cluster segmentation of the weighted undirected graph, the graph clustering algorithm includes but is not limited to the chameleon clustering algorithm, the Graph Cut algorithm, and the Felzenszwalb algorithm, and the present application does not impose any special restrictions on the type of graph clustering algorithm.

[0104] 207, based on the classification results, determine the sample sampling order and the alternating order between buckets during the model training process, and complete the training of the U-Net model based on the sampling order and the alternating order in combination with additional embedding conditions.

[0105] Here, based on the classification results of the training set of each style, the sampling rules of the images in each bucket during U-Net training are first determined; secondly, the alternation order between buckets is determined based on the clusters where the corresponding nodes of the images between buckets are located. In this way, through the subject similarity between images, images with similar content and structure in each style are sent to U-Net for training in an orderly manner, while enhancing the learning ability of the subordinate relationship between attributes and objects in the image.

[0106] Specifically, Stable Diffusion XL is a two-stage diffusion model consisting of a cascade of a base model and a refiner model. The base model is used to obtain the Latent features of the image, and the main task of the refiner model is to denoise and improve the detail quality of the Latent features to obtain refined Latent features, which are then processed by the decoder to output the style image.

[0107] During each U-Net training process, a preset number of images belonging to the same category are sampled in each bucket as training samples, and the size of the preset number is equal to the batch in the preset parameters of the U-Net model; and when switching between buckets, the set of images corresponding to the cluster center nodes of the clusters to which all samples in each bucket belong is used as the category representative set of each bucket; the similarity C between the category representative sets of the two buckets is calculated. In this embodiment, the Jaccard similarity coefficient between the category representative sets of the two buckets is calculated as the similarity C. The Jaccard similarity coefficient is a well-known technology and the specific process is not repeated. During the training process, the sequence obtained by arranging the similarity C between the category representative sets of the current bucket and all other buckets in descending order is used as the alternating sequence sequence of the current bucket. When the sample sampling is alternately exchanged between buckets, it is performed in the order of the alternating sequence sequence, that is, the bucket corresponding to the first similarity C in the alternating sequence sequence is selected as the replacement object of the current bucket.

[0108] It should be noted that this embodiment only provides a method for calculating the degree of similarity between category representative sets, that is, calculating the Jaccard similarity coefficient between category representative sets. On the premise that the degree of similarity between category representative sets can be evaluated, in other embodiments, other calculation methods can also be selected to obtain the degree of similarity C respectively, and this application does not impose any special restrictions on this.

[0109] Furthermore, according to the above steps, the content-structure feature vector of each image in each style training set is obtained respectively, and Fourier encoding is performed respectively to obtain the Embedding feature corresponding to each image, and the Embedding feature is embedded into Time-Embedding. The embedding is an add operation, and is input into the U-Net of the base model as an additional style image structure condition. The U-Net is trained to allow the model to learn the layout structure features and image content under different styles. Secondly, combined with the sampling rules of the samples in the bucket during the above-mentioned U-Net training and the replacement order between buckets, the U-Net training is completed. The loss function of U-Net is the cross entropy loss, and the optimizer is Adam; secondly, the output of the U-Net model after training is sent to the refiner model for denoising and decoding, and finally the style image is output to complete the training of the SDXL model. The training of the neural network is a well-known technology, and the specific process will not be repeated.

[0110] The generated model SDXL is then deployed on the server so that the AI ​​imaging device can communicate user needs with the server at any time. Deploying the trained generated model SDXL on the server can also save the volume of the AI ​​imaging device. On the other hand, the server can communicate with AI imaging devices in different locations within the communication range, so that the AI ​​imaging device can better provide style images.

[0111] 208, the AI ​​imaging device communicates with the server according to the user's imaging requirements, generates a style image using the trained SDXL model, and sends it to the AI ​​imaging device.

[0112] Specifically, when the AI ​​imaging device receives the user's personal photo, it generates a style image that meets the user's needs in combination with the style type selected by the user. It should be noted that the AI ​​imaging device receives personal photos in ways including but not limited to the user sending his or her own photos to the AI ​​imaging device via Bluetooth or other methods, and can also use the AI ​​imaging device to take personal photos in real time.

[0113] The user will select style-related information on the AI ​​imaging device by clicking or inputting according to personal needs, such as clicking on the ancient style module, selecting the seaside environment, etc. The AI ​​imaging device sends the style-related information and personal photos selected by the user to the server at the same time, and then uses the SDXL model deployed on the server to generate a style image that meets the user's needs. It should be noted that the more style images SDXL generates, the more results are available for users to choose from. Therefore, under the condition that computing resources can be met, SDXL can generate as many style images as possible, and the server returns the generated style images to the AI ​​imaging device, and displays them on the AI ​​imaging device for users to choose. Finally, the AI ​​imaging device sends the style image selected by the user to the user terminal, and the user terminal includes but is not limited to mobile phones and tablets.

[0114] In the second aspect, a multi-style image generation device based on a large model is provided, such as Figure 5 As shown, the device 500 includes:

[0115] A preprocessing module 501 is used to construct a training set for each style using the de-noised images in each style;

[0116] A first extraction module 502 is used to divide the images in the training set into positive and negative samples, and obtain a deep feature vector of the content of each image using a contrastive learning model;

[0117] The semantic extraction module 503 is used to take the training set of each style as input and use the semantic segmentation network to determine the true label of the pixel point in each image, the semantic label vector of the feature point and the structural feature atlas of the attribute;

[0118] A second extraction module 504 is used to determine a structural feature vector of a feature point based on the semantic label vector of the feature point and the structural feature atlas using a graph attention network;

[0119] A stitching module 505, configured to determine a content-structure feature vector of each image based on the deep feature vector and the structure feature vector;

[0120] A classification module 506, for classifying the training set using a graph clustering algorithm based on the similarity of images in the bucket during the training of the U-Net model in the SDXL model;

[0121] A training module 507 is used to determine the sample sampling order and the alternating order between buckets in the U-Net model training process in the SDXL model based on the classification results, and complete the training of the U-Net model based on the sampling order and the alternating order by using the content-structure feature vector as an additional embedding condition;

[0122] The imaging module 508 is used for the AI ​​imaging device to communicate with the cloud according to the user's imaging requirements, and uses the trained SDXL model to generate a style image and send it to the AI ​​imaging device.

[0123] It should be noted that: the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the multi-style image generation device based on a large model provided in the above embodiment and the multi-style image generation method embodiment based on a large model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0124] In addition, in one embodiment of the present application, the AI ​​imaging device may include a processor, a memory, a touch screen, a camera, a fill light, a thermal sublimation printer, a mainboard, a wireless communication module, a wired communication interface, mechanical components, and a power supply. Among them, the memory stores executable program code and temporarily caches user personal photos; the touch screen is used for users to select style-related information; the camera is used to obtain user personal photos; the fill light is used to provide light source; the thermal sublimation printing plate is used to output the style image received by the AI ​​imaging device; the mainboard is used to control and manage various functions and operations of the AI ​​imaging device; wireless communication modules, such as Bluetooth and Wi-Fi, are used for the AI ​​imaging device to communicate with the server or with the user's mobile terminal; the wired communication interface is used for the transmission of wired data; the mechanical components are used to adjust the position or angle of the device; the power supply is used to provide the power supply required by the AI ​​imaging device.

[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-style image generation method based on a large model, characterized in that: The method comprises the following steps: Use the de-noised images in each style to build a training set for each style; The images in the training set are divided into positive and negative samples, and a deep feature vector of the content of each image is obtained using a contrastive learning model; Taking the training set of each style as input, a semantic segmentation network is used to determine the true label of each pixel in each image, the semantic label vector of the feature point, and the structural feature atlas of the attribute; Determine a structural feature vector of a feature point based on the semantic label vector of the feature point and the structural feature atlas using a graph attention network; Determine a content-structure feature vector for each image based on the deep feature vector and the structural feature vector; Classifying the training set using a graph clustering algorithm based on the similarity of images in the bucket during the training of the U-Net model in the SDXL model; Based on the classification results, the sample sampling order and the alternating order between buckets in the U-Net model training process in the SDXL model are respectively determined, and the content-structure feature vector is used as an additional embedding condition to complete the training of the U-Net model based on the sampling order and the alternating order; The AI ​​imaging device communicates with the server according to the user's imaging needs, and the server generates a style image using the trained SDXL model and sends it to the AI ​​imaging device.

2. The method for generating multi-style images based on a large model according to claim 1, characterized in that: The step of dividing the images in the training set into positive and negative samples and obtaining a deep feature vector of the content of each image using a contrastive learning model includes: For each style of training set, use the image segmentation algorithm to divide each image into multiple image blocks, and use all the image blocks of one image as positive samples, and use all the image blocks of the remaining images as negative samples; The positive and negative samples are taken as input, and the feature vector output by the contrastive learning Moco model is used as the deep feature vector of the content of the image corresponding to the positive sample.

3. The method for generating multi-style images based on a large model according to claim 1, characterized in that: The training set of each style is used as input, and a semantic segmentation network is used to determine the true label of the pixel points in each image, the semantic label vector of the feature point, and the structural feature atlas of the attribute, including: Taking the training set of each style as input, using the semantic segmentation network to obtain all semantic labels of each pixel in each image in the training set and the confidence of each semantic label, and taking the semantic label with the highest confidence as the true label of each pixel; Pixels with the same true label in each image in the training set are regarded as a class of pixels with the same attributes, a connected domain of each class of pixels is determined, all feature points in the connected domain are determined using a feature point detection algorithm, and a vector composed of semantic labels at all confidence levels of each feature point is used as a semantic label vector of each feature point; The structural feature atlas of each attribute is determined based on the connected domain extraction results of the feature points.

4. The method for generating multi-style images based on a large model according to claim 3, characterized in that: The method of determining the structural feature atlas of each attribute based on the connected domain extraction result of the feature points includes: The undirected graph formed by connecting all feature points in each connected domain in each image is taken as the connected structure graph of each connected domain, and the set of connected structure graphs of all connected domains under each attribute is taken as the structural feature atlas of each attribute.

5. The method for generating multi-style images based on a large model according to claim 4, characterized in that: The method of using a graph attention network to determine a structural feature vector of a feature point based on the semantic label vector of the feature point and the structural feature atlas includes: The semantic label vector of each feature point in each connected domain is used as the initial feature vector of each feature point in the connected structure graph of each connected domain; The connected structure graph after setting the initial feature vector in the structural feature atlas of all attributes in each image is taken as input, and the structural feature vector of each feature point is obtained using the graph attention network.

6. The method for generating multi-style images based on a large model according to claim 5, characterized in that: The method for determining the content-structure feature vector of each image based on the deep feature vector and the structure feature vector is: A set consisting of structural feature vectors of all feature points in each connected domain in each image is determined, and multiple sets consisting of structural feature vectors of all feature points in all connected domains in each image are determined; In the order of the deep feature vector about the content of each image first and the multiple sets later, the deep feature vector about the content of each image and the concatenation result of the multiple sets are used as the content-structure feature vector of each image.

7. The method for generating multi-style images based on a large model according to claim 1, characterized in that: The similarity of the images in the bucket during the training of the U-Net model in the SDXL model is used to classify the training set using a graph clustering algorithm, including: The training set of each style is bucketed based on the aspect ratio; The similarity measurement result between each image in each bucket and any image in another bucket is taken as the first similarity; The similarity measurement result of the content-structure feature vector between each image in each bucket and any image in another bucket is used as the second similarity, and the ratio of the second similarity to the first similarity is used as the main similarity between the two images; Each image is taken as a node, the subject similarity between images is taken as the weight between nodes, the undirected graph constructed using all images in the training set of each style is taken as input, and the graph clustering algorithm is used to obtain the clustering cluster where the image in each bucket is located.

8. The method for generating multi-style images based on a large model according to claim 7, characterized in that: The method of determining the sample sampling order and the alternating order between buckets in the U-Net model training process in the SDXL model based on the classification results includes: During each training process of the U-Net model, a preset number of images belonging to the same cluster are sampled in each bucket as input for training; When switching between buckets, the set of images corresponding to the cluster centers of the clusters to which all samples in each bucket belong is used as the category representative set of each bucket; Calculate the similarity between the category representative sets of two buckets, arrange the similarity between each bucket and the category representative sets of other buckets in descending order to obtain a sequence as the alternating sequence sequence of each bucket, and when sampling is alternately exchanged between buckets, follow the order in the alternating sequence sequence.

9. The method for generating multi-style images based on a large model according to claim 1, characterized in that: The training of the U-Net model is completed based on the sampling order and the alternating order using the content-structure feature vector as an additional embedding condition, including: Perform Fourier encoding on the content-structure feature vector of each image to obtain the corresponding Embedding feature, embed the Embedding feature into Time-Embedding, and input it into U-Net as an additional style image structure condition; Time-Embedding is used to train the U-Net model based on the sampling rules within each bucket and the alternating order between buckets.

10. A multi-style image generation device based on a large model, characterized in that: The device comprises: A preprocessing module is used to construct a training set for each style using the de-noised images in each style; A first extraction module is used to divide the images in the training set into positive and negative samples, and obtain a deep feature vector of the content of each image using a contrastive learning model; The semantic extraction module is used to take the training set of each style as input and use the semantic segmentation network to determine the true label of the pixel points in each image, the semantic label vector of the feature points, and the structural feature atlas of the attributes; A second extraction module is used to determine a structural feature vector of a feature point based on the semantic label vector of the feature point and the structural feature atlas using a graph attention network; A splicing module, configured to determine a content-structure feature vector of each image based on the deep feature vector and the structure feature vector; A classification module, for classifying the training set using a graph clustering algorithm based on the similarity of images in the bucket during the training of the U-Net model in the SDXL model; A training module, for determining the sample sampling order and the alternating order between buckets in the U-Net model training process in the SDXL model based on the classification results of the classification module, and completing the training of the U-Net model based on the sampling order and the alternating order by using the content-structure feature vector as an additional embedding condition; The imaging module is used for the AI ​​imaging device to communicate with the server according to the user's imaging requirements. The server uses the trained SDXL model to generate a style image and sends it to the AI ​​imaging device.

Images